If you put in the work, inside and outside of college, then AS Physics should not be too hard. However, the fact that the principal is saying that nobody has gotten more than a C suggests to me that the teaching may be below par.
One of the main things to consider is whether you enjoy or are interested in the subjects. You certainly seem interested in physics, and combining it with maths should aid you in your journey towards working in finance.