You would be competing for a place against people who are taking or have taken three A levels or the IB all in one go, not two A levels and then one. You might place yourself at a competitive disadvantage by doing two and then one.
Do you wish to drop Physics because of the workload, or because you do not like and/or are not doing well in Physics? If workload is the issue, Oxford might not be the right university for you. The undergraduate workload at Oxford is not as extreme as some claim, and there is still time for rest and fun, but the workload is quite demanding.