Though not entirely the same, I did a resit year for my A-levels and found it boring as hell. This was mainly because I didn't know the people in the year below, so it was a bit awkward to begin with. If you know people in the other college then this might not be as much of a problem.
I don't see why a university would object to it, but it would still be worth contacting a couple for their take on it. Physics isn't always required, though some universities will ask for applied modules in mechanics to offered as part of the maths course - That said, if you're doing further maths, I'd assume you're looking to the higher rated universities where it may be required. Plus, you'd only have two A-levels if they didn't want the physics so you'd have to carry on one of the other things.
The other option, depending on how you feel about it, is self-study for the physics. The college should still be able to provide papers for this (I think you'd go in as an external candidate if it's a course not typically offered) and you wouldn't have to move anywhere. It depends how you feel about the physics and if you could handle it by yourself. Do you need it to be taught? The only consideration would be the practical exam if your exam board requires this. I think most do though, it would just be the same as the AS practical but for A2. Still, I don't see why a teacher would be unable to assess that exam if they're doing it for the AS students.
In short: Contact the universities, just get their verdict. If you feel capable of teaching yourself the course, then ask the college about the exams. If you don't feel happy about that, then switch college.