There are a number of factors that likely contribute to your own level of self-esteem. Many clients have identified their preteen and teen years as the point at which they began to be aware of having "low self-esteem" and struggled to gain it back.
Parents, teachers, peers and other important people in our lives influence the shaping of our self-esteem and confidence. I think because those early years are the age in which we are still shaping our identity, our core self is not as stable and thus more easily wounded or bolstered by those we care about.
But I also think it is important to be aware that self-esteem is something that continues to grow and change throughout life. So, if you feel like your confidence is low right now, it is not a life-sentence but a point from which to make some changes and begin to live in a way that is focused on wholeheartedness. This concept comes from Brene Brown who does research about shame and vulnerability. She has an amazing Ted Talk about it. Check it out:
Pretty powerful right?
A new boyfriend or girlfriend
A compliment from your boss
What do these things have in common? The ability to temporarily improve your self-esteem. When validation comes from something outside of you, it is an external validation and it doesn't tend to last. The ability to validate yourself and soothe yourself from negative interactions and feedback has to come from within - internal validation.
The process of improving your self-esteem is certainly not easy but well worth it. I like to use a number of cognitive behavioural approaches with clients. We talk about family of origin and early memories of self-esteem.
If you are interested in doing some of this work on your own, I recommend reading The Gifts of Imperfection by Brene Brown who offers a number of different "guideposts" about living in a way that she calls "Whole Heartedly." The image in this post comes from her website.
I also recommend tapping into some creativitiy - that might be anything from painting to making music to baking and beyond. Creativity is a form of expression but also taps into something that feels authentic.