Yes, it's possible. Physics uses maths extensively, but it's entirely possible to lack the physical intuition to do well in physics - particularly in the "lower levels" in school and perhaps early undergraduate physics.
If you continued in mathematics you would still probably do "well" in higher physics, if you so chose, where mathematical formalism can be more important (and in some cases is fundamental to the actual topic at hand, which leads many who have a stronger physical intuition to struggle with those topics...)