The thing is that physics/maths isn't like "macaroni and cheese but don't like macaroni". It's like "pasta, with anything, but have a wheat intolerance". Like...physics is fundamentally entwined with maths. You WILL have to do maths, to do physics. They aren't separate entities - realistically physics is "just" applied maths.
However, the question of whether you need to do A-level Maths with physics, is another issue entirely. If you intend to apply to architecture, then you could reasonably take physics without mathematics, and you wouldn't find yourself held back by that in applying (but may struggle more with the physics course due to having less constant exposure and practice of the basic maths used). However, you'd get just as useful preparation in mechanics, but much better preparation in it, by taking A-level Maths. Plus, you will need to do maths consistently throughout A-level Physics, which will be a the "high end" of GCSE level maths. So...if you hate maths why are you considering something...which just makes you do maths...?
I'd probably suggest doing History, as you'll need to write essays in an architecture course to some extent (for theory/history modules/topics) and if they require Physics, they probably also require Maths - and I'm fairly certain very few require either. Thus, they will expect to be teaching you any relevant aspects of the subjects from scratch anyway (whereas they're more likely to leave "learning to write an essay" as an exercise for yourself - which can be a bit tricky to trial and error your first couple essays). Also a lot of the 20th century architecture was shaped by/developed in response to modern historical events, such as the world wars/cold war and the associated technological and social changes. So you would have some nice (albeit debatably useful) context for the above.