I was under the impression that it isn't possible to take all three sciences in the IB? Apologies if this has changed since I finished college, but I remember this being one of the reasons why I chose to do A levels instead.
The admissions requirements for vet schools can be found on their websites; only biology and chemistry are a requirement. Taking physics (or maths) does not confer any advantage with regard to appearing a more academically able applicant. There is a small amount of physics in the veterinary course, but nothing that you will struggle with if you don't take IB physics, and conversely there are subjects that will help you better with other aspects of vet school (i.e. essay writing). I personally adored doing physics in college and it was my favourite subject, so by all means do it if you enjoy it, but don't be under the illusion that it makes you more likely to get in.