Neither, both completely irrelevant. So pick whichever you think you'll get a better grade in.
There is some A-level Chemistry content that is useful for/relevant to a physics degree, although it's a fairly small amount and most is irrelevant, and it's never assumed before the degree commences either so you can live without it.
Of course, A-level Chemistry is commonly required for certain other courses like chemistry, chemical physics, medicine, etc, so gives you some more options in those areas if you do change your mind later. It's also useful background for engineering, particularly chemical or materials engineering (although not usually required if you have physics).
But for maths specifically there is really no particular relevance of either.