Lots of harsh comments here. At Cambridge you would be expected to utilise good maths skills, certainly. But it also wouldn't be coupled with 5 other subjects plus all the other crap the IB involves. The fact you have HL physics will prove to the university that you do have the ability to utilise applied mathematics. If you can justify why to the university under interview pressure you made that decision INTELLIGENTLY (i.e- not 'I found it hard', but rather "I felt Physics would show my abilities, and I therefore utilised the last HL slot to pursue something of interest to me" ) then it would be OK.
That said, you're alternative solution doesn't sound sensible. Maths SL isn't an easy jump from GCSE- it's just easy compared to HL. Without experiencing the additional tuition from HL maths, SL will seem difficult and consume just as much time as HL. You wouldn't realistically be able to take on Further Maths given the time you will have to dedicate to the IB. HL maths will be your safer option. It's hard, but my school had at least 1 level 7 this year (out of 6 students) so it's not impossible.