I'm not expert, but let me tell you my experience. I was brought to conduct sexual things with other boys at very young age. I'm feeling like straight but when I'm aroused my interest changes. I used to watch gay porn, a lot. In one way that negative habit brought me here, which was actually surprising. I used to watch porn sites regularly but with time I've started to look for gay porn more than other. I looked for amateur videos and somehow I recognize myself trough watching it. In my life I was sexually connected in full meaning to just one male person - my own twin brother and it was not let say abuse in real terms. We were brought to sexual behavior by other older boys (that was abuse) and went further with "tide". In different movies I've recognized men who were brought to sex early, or forcefully and conclude that by their sexual behavior. I saw that such persons have tendencies to pushed their boundaries. This is something that brought those people to very dangerous, wrong and destructive behavior. I've recognized all those forces in me, actually I was looking for such details, signs, experience etc. I've discovered that I'm the same. I was lucky as I was always scared to let myself completely to my inner demons. By watching other people I've learned that I could be heading for some let say dangerous situation and could not be able to stop. Now I'm somewhere at braking point, I'm completely aware of things that are imprinted in me. Sometimes I'm not strong enough to resist, I'm very careful during my stress times, those are times when I'm the most vulnerable. I've learned that dreams are healthy if they are just dreams/fantasies. Other thing is that fantasies which are connected to fear are always stronger imprinted in our mind. This is important mechanism for our survival; scary and threatening things we should immediately remember.
Even you were abused just one time, abuse is abuse and this can have impact on your later life. Certainly you should find source of your fantasies, I'm not sure by what kind of method, keep looking.