Physics is generally considered one of the harder A levels, but so are chemistry and to a lesser extent, biology.
A lot of people will tell you that you 'need' to take A level maths alongside physics, but there is very little mathematical content in A level physics in order to make it accessible to those not taking A level maths. There will be some exponents and logarithms questions, but these topics aren't difficult to pick up if you did well in maths at GCSE.
The reason why taking maths is recommended is because A level physics has a lot of formula-rearranging, standard form, proportionality and the need to determine how a quantity changes when one variable changes in the equation. None of this is beyond GCSE level, but you do need to be very comfortable with these skills to do well. In addition, the mechanics topics in A level maths complement the physics mechanics topics very well.
If you're taking 3 A levels and planning to go into medicine, perhaps it may be a better idea to pick something else other than physics (although if your basic maths skills are very good, physics should be doable) if you aren't planning to take maths (to make it more likely that you'll get the grades). Have you considered biology, chemistry and maths?