Some amount of homoerotic horseplay seems to be almost unavoidable among brothers within close age. The issue is always whether a person is pressured, harmed, threatened, or kept from refusing / ceasing / leaving if and when they wish. I could see that story being harmless experimentation or a source of nightmares based entirely in the mental state of the people involved. If it disturbs you then it was wrong - you need no further justification than that (*). Now, it is absolutely possible your brother saw and felt nothing wrong and had / has no idea of any bad effect on you (if there is one). You can steel yourself up to bring it into conversation:

"You may not remember this but there was one particular incident between you, me, and Jim that really disturbed me. I felt he took advantage of me when I wasn't in a condition to understand enough to say no. And I'm honestly concerned about the safety of kids near him. Maybe you think I'm wrong but that's how I feel about it, and I just hope you'd give the issue thought before letting Jim around your son in a private setting. At least make sure the boy knows all the 'bad touch' issues. It really isn't worth the regret."

(* = I saw a fairly graphic example of this as a boy. I was 13 and so was my best friend at that time. His kid brother was 11. Without going into too much detail, the older brother made it very clear that they would now play their apparently usual dry-humping game - he called it "Prom Date" - right then and there. Younger brother objected to doing it *now* but older would not and did not take no for an answer. Younger's complaints and attitude, looking back now, suggest that this was something he was okay with (or liked) most of the time but he was distraught and humiliated at being made to show it in front of another older boy (me) in what turned into a very much of a sexual dominance display - really apparently for older brother to show off to me what he could do for fun. Younger brother several times shot me looks of pure unmistakeable anger - I didn't belong there, it was different with me there. Apart from that one incident I was with both of those brothers regularly hanging out for years and years and they seemed like normal brothers on good terms. Older brother was EXTREMELY sexually aggressive / suggestive basically at all times. He introduced me to some behaviors that I enjoyed at the time but there was no mistaking that it was always his initiation / insistence. Years later I ended our friendship when he raped his girlfriend and then forwarded me the email she'd sent him detailing her shock and confusion and not knowing how to go on with life - like it was a joke he wanted to share with me!! But P.S. she got back with him and married him. World is a messed up place.)
My story

"Don't think it hasn't been a little slice of Heaven just because it hasn't!" --Bugs Bunny