I have this problem, too. I never connected it to self-worth, but rather with emotional dysregulation.
Like something that happened to us made it impossible for us to gauge how much emotion to use in a given situation. So, we bottle it up and use no emotion or uncork it and it goes to 100%. No in-between.
Maybe emotional regulation is connected to self-worth? I don't know. I just know that I don't trust my emotions until someone else confirms or corrects me.
I'll be just fine and dandy
Lord, it's like a hard candy Christmas
I'm barely getting through tomorrow
But I won't let sorrow get me way down.