I didn't do physics at AS, I did geography instead, but I do study further maths. However the offer is an A in maths OR further maths, so it doesn't seem to matter.
As long as you're good at the arts subject and you do well, your application should be just as strong. Check the colleges though because bean87 is right, some appear to prefer candidates with more sciences.
Edited: Also I agree with Vazzy B - you will probably have to drop the arts subject for A2. And anyway taking an art subject is useful as it will give you essay practice - good for BMAT!!