You need support and professional help. I have learned anyone that violates a person or persons property or sense of freedom, inflicts any form of unwanted physical contact and has tendency to anger that damages a person or property needs help. I experienced and understand, for me it triggered the past. For your brother he saw the damaged person but fear will keep him away. I have discussed these type of actions with professionals and they strongly recommend counseling for those who undertake these actions. Sadly many do not see the need for help and surround themselves with others who believe these actions are acceptable and this creates a vicious cycle of unhappiness for all. There is no shame in seeking help, we all carry issues and wounds, and seeking help is a sign of strength. You want your brother back and rightfully he knows your actions are a cry for help. Now you need to find a way to heal so he can begin to see the good in you. Now he has fear--fear is not always dependent on size of the person but rather the actions, the emotions and expressions that occurred during the act and how others react to these acts. Fear will keep him away and distant from you. I have fears and will never allow others to put me in a position to trigger the abuse.
Please seek help because you are aware of your actions and outcome. The underlying rage was triggered from your past abuse and/or other emotional issues that you harbor.
I wish you well and I can tell you facing your past will allow you to heal and build a life you deserve.