In all honesty maths is one of those 'useful for most things' subjects that I would recommend taking for AS (As AS maths isn't difficult) as it shows that you can logically think through a problem step by step and solve it.
Physics AS is fine without maths I would say, just being able to rearrange and equation and plug numbers which should be fine from GCSE.
Depends what course your school is doing, as they cover different ideas. Some physics courses contain a little bit of mathematical modelling which might be a bit awkward if you don't practice using maths much.