I have this problem, too. I never connected it to self-worth, but rather with emotional dysregulation.
Like something that happened to us made it impossible for us to gauge how much emotion to use in a given situation. So, we bottle it up and use no emotion or uncork it and it goes to 100%. No in-between.
Maybe emotional regulation is connected to self-worth? I don't know. I just know that I don't trust my emotions until someone else confirms or corrects me.
But he grew old, this knight so bold / And upon his heart a shadow / Fell as he found / No spot on the ground / That looked like El Dorado.