I have a friend who does Physics - arguably the most mathematical of the sciences - without having taken Maths to A-level. Now, admittedly, he is the only kid in our higher physics set to do this, but he's probably going to come out of it with an A. In fact, there's specific notes in the back of the textbook for students who didn't do Maths to A-Level
What I would make sure is that, if you have your eye on a degree/university already, PLEASE check what they require for you to go and study there. I think I'm right in saying that Maths at A-Level is the most 'required' subject for various degrees... definitely a physics degree