I wouldn't call the arts or social sciences 'dumb subjects' and I'm a STEM man.
While its true that they don't require the same cognition that the sciences do they still reflect their own form of intelligence. I would call the sciences the best measurement of raw mental capacity but I know people who get A*A*A* who if you asked them to come up with an interpretation of literature would be completely at a loss and who still think a comma is for "when you take a breath".
(I do think social sciences get a bad wrap. Personally, I have a respect for these subjects but I think they're horribly oversubscribed and misdirected. This is all off topic though)
If you didn't get a B in GCSE maths then I'd say STEM isn't for you because the maths content will get harder not easier. Maybe you can work something out with regards to resitting your GCSE maths but idk if that would be possible since you can't take a gap year.
EDIT: So you got a B in most things but not maths. Maths is pretty fundamental to physics and if you struggle with maths you will struggle with physics. You'll just have to explain to your interviewer. If you indicate a real vocation then they might let you do it but if your reasoning is "I don't want to do dumb subjects" then I wouldn't hold much hope.
At my college they made everyone who did physics do maths as well because physics really does involve a lot of maths.