I'm a gay man who isn't transgender. I have some friends and colleagues who do identify as transgender. I've also spent several years worked on gender identity issues in the workplace.
It sounds like you would benefit from talking with a therapist who is familiar with gender identity dysphoria, and from reading some personal accounts of both men who were raised as girls for some period of their lives, and men who have come to feel that their true gender identity doesn't match their birth gender. There is a broad spectrum of gender identity and you don't need to live only at the polar ends.
I'll do some research in the next couple of days and see what resources I can assemble for you.
So many of us have challenges figuring out if our sexual orientation comes from within ourselves, or whether something in our experience might have imposed our desires upon us. Your dilemma feels very familiar to me.