The offer from Oxford for physics is A* A A. You have achieved one of the As. Therefore you still need to get an A* in one of the remaining 3 A levels and another A. Personally if it was me I would throw the energy into the remaining 3 as well as the entrance exam preparation. If you dont do further maths when it was possible to do it in your school they will want to know why you dropped it. As you achieved the required grade standard a year early I think you are doing well, personally so it depends whether you think you would have potential to get at least one A* out of your other subjects. I would read the commentary from Oxford as being geared towards those who don't avhieve the required grades a year early but you have (as long as you think you can get an A* in one or more of the remaining 3 as well as a further A). If you cant you may not be at the standard to pass the entrance exam anyway. Further Maths is supposed to be useful for the physics degree which is quite mathematical at Oxford so, if you are progressing well in it, it would be a shame to drop it just to try and improve maths A level up to an A*. You are going to need to prepare for the entrance exam which includes a lot of maths so they will be looking at how you perform in that and it will probably develop your maths skills further anyway.
Another possibility might be for you to resit the maths A level along with the other 3 next summer to see if you can improve it then. Since you are doing further maths you are already continuing to work on your maths skills and will also be doing this to prepare for the entrance exam (I am assuming you will be making use of the preparation materials on the Oxford Uni website). Therefore it may not be too much extra work just to keep going over the maths A level syllabus and doing practice papers before resitting it with the others although it will detract a bit from your efforts on the other 3.