Physics and maths compliment eachother. But maths probably compliments physics more than vice versa, and maths is a good all rounder as well as offering support to pretty much any other subject that uses numbers. Put simply, its a good idea if you are going to do physics, do maths as well. If not, its your call whether to do maths or not. It makes more sense to do maths without physics than physics without maths