There are times when I seriously feel like you, but I've come to realise that shallowness/image-obsessed society can be society's problem - I'm going to have a good time and be happy anyway.
I mean, what does being ugly actually stop you from doing? I'm sure you have friends, you said you've had relationships, you can still eat anywhere, go to any club, get any job.
There are disadvantages - there's more pressure to have a decent personality, you might not get the prettiest girl (but looks are overrated anyway!) and the mirror's never going to be kind, but there are actually upsides.
All the shallow, vain people are automatically filtered from your friends and relationships (this is an important one - my pretty friends never know if they're being used for their looks, whereas I know I'm liked despite my looks), and, it sounds harsh, but there isn't that same pressure to look good/okay all the time, because people just know you're ugly.
Also, if you have a good personality, people will be surprised, and find you even more charming. The first impression of attractive people is witty and cool, but if they don't live up to it (fairly often) we're disappointed. The default setting for me is "ew" or "bleurgh", so I see the surprise when people realise I'm funny, talkative and sociable.
The biggest mistake you can make is letting "ugly" run your life. No-one calls themselves ugly without reason, I know this myself, but it's begins and ends with the way you look.