I didn't go to my Y11 prom and don't regret it at all. I hated my school, hated the people and apparently it was rubbish.
I understand your anxieties surrounding your prom. I changed school for sixth form and throughout the whole of Y12 and 13 I said I wasn't going to go for the same reasons you aren't going, I didn't want to be around people I didn't care for and even worse, being an external student I didn't know very many people. I ended up changing my mind the *week* before and I actually loved it. Everyone was really friendly and nice, even people I wouldn't usually speak to or didn't even know, so the choice really is yours, but I'm happy I went in the end but I'm sure I would've been just as happy not having gone.