I'm a Cambridge Maths student and I didn't take Physics as my others were Chemistry and French, but I have a friend whose other A-Level was Music. I can only speak for Cambridge but it really is just your Mathematical ability that they care about and Maths and FM A-Level (alongside any entrance exams you take) are generally sufficient.
When it comes to taking Physics modules, I don't know how that will affect it, as, at least at my uni, the more applied courses are taught in a purely Mathematical sense, i.e. the Physics is basically just abstracted away and I'm focusing on the calculations. There is no option (at Cambridge, there may be at other unis) to take the same Physics courses that the Physics students (NatScis) take other than in the first year, but that is a specific option you apply for (which may require Physics at some colleges). In general, I would say that not having Physics at A-Level has not disadvantaged me whatsoever.