I can legit tell you that in over 100 years of medicine between us, the straw poll of medics I have just done have never used anything beyond GCSE Physics! And even then not anything you couldn't learn in 1 lecture at med school. Trust me, the med stats is much more challenging, but nobody suggests you must do statistics!! (an anaesthetist, consultant medic, radiologist and GP).
If you enjoy Physics, by all means do it, but not because you think there is a lot of it in Medicine!