Triggers are triggers, regardless of what...or who..they are.
I was molested and physically tortured by my father who mostly identified as an exemplary role model of an evangelical church leader.
My triggers range from church potlucks and Sunday School rooms to medical exams and Mennen aftershave. I panic when I get an invitation to a party. A PARTY, for god's sake! To me even a party is a confined space with a group of people, much like an elevator...as interpreted in the back of my brain.
You're not "out there" at all. As others mentioned, it's just one of your triggers. In my situation, I've learned that they grow stronger when I fight them, so I've learn to manage them instead.
I have taken to the stand-up comedy stage to educate other male survivors and those who try to love them. I blog about my isolated religious upbringing where physical and sexual abuse were commonplace and I serve as a facilitator of a weekly support group for men who have suffered sexual assault.