Put it this way: I got an A at GCSE Maths, then an E in Maths and Physics at A Level. I swapped to English Lit and achieved an A, predicted A* at A2. You have to play to your strengths at A Level, and if you're worried now that your Maths is a weakness then I'd think very carefully before going into Physics.
If you can really dedicate the time to learn the maths of it, and you love it as other poster said, then ok- but exam papers tend to throw a lot of maths questions at you and very little theory.
If you have good teachers and you really want to then go ahead. Best of luck eitherway.