What do you mean you don't have another option? Seeing as you have taken the compulsory subjects required for medicine you can essentially take any 4th subject you want - physics, economics, english, geography, etc.Just choose the one you enjoy the most.
As to physics being the hardest AS subject there is, from what I've heard it is a significant step-up from GCSE but, like any subject, if you put in the hard work you'll end up doing fine, so I wouldn't discount physics on that basis.