Not in the slightest mate. I changed A-levels, which meant I had a year extra, went to uni straight afterwards just coming up to 20. Loads of stuff happened at home which I won't go into. I also wasn't enjoying where I was so I dropped out.
Luckily you're allowed a year where you drop out, 4 years worth of funding, so I went back at... just short of 23? Had the best 3 years of my life, met a partner for a few years.
Now I am 29 coming up 30 and about to start my Masters. Would've gone straight on to do it but it's been a bloody hard journey the years in between.
Don't worry about your age. You won't feel left out and if anyone tries to make you feel left out for a couple years difference they're just one of those idiots who'll always try and find fault in you regardless. Who find fault in others all the time.
Just try your best to do a masters or grad job afterwards ASAP. Not because it's wrong to put of doing one, but cos the years can easily fly by and you want to aim to put yourself in to as good a position as possible.
But no shame in taking a year or two between to save if needs be.
Best of luck mate!