Does your college offer an option between human and physical geography at A2??
If so, I'd say the physical geography stuff is definitely sciencey, the majority is learning processes like plate tectonics and weather and climate, but you do still have to do a bit of human geography. I chose the human geog stuff (i.e, development and globalisation, etc) because even though I do maths I don't do Biology/Chemistry/Physics and find just learning processes and sciencey stuff quite boring, so yeah I see some aspects of geog as rather science based.
Anyway, if you enjoy something you should continue it, who cares if it isn't 'sciencey', to be honest some unis even count geography as a science.
You haven't even done the AS yet, so you don't need to decide just yet, you never know one of your other subjects might not go as well as you hoped and you might end up wanting to drop one of those instead.