A guy in my college does this, but he did maths at AS, dropped it at A2. I think he would've found it really difficult if he hadn't done maths at AS, as there are a lot of concepts that overlap. For instance, simultaneous equations to a higher than GCSE level,
certain kinds of graphs (that you have to understand and know how to draw), trigonometry, and most importantly the sheer amount of equations you need to know in physics! You get a formula sheet full of equations! You have to know how to apply them, and how to rearrange them, which maths helps with. Physics is definitely so much easier with maths.
I don't think you need to do it to A2, but you need at least a baseline knowledge.