It would be risky to reject your offer. The acceptance rate for medicine has hovered around 16% for the last few years, and the difference between successful candidates and those who are rejected is often wafer-thin - there's no guarantee you will find yourself in that 16% another time. You've done extremely well to get in, and if you want to be a doctor, you'd best take the opportunity.
Keep an open mind. Democracy is right. St Andrews is a really beautiful place, and you'll be spending half your course in a much bigger city anyway. From a medicine point of view, that might give you some interesting insights into the difference between rural and urban practice. From a social point of view, it can be nice to experience different things, including things you might not have discovered otherwise. You may find you really like it in Fife! Yes, there are stereotypes associated with certain universities, but stereotypes work like magnifying glasses - they exaggerate certain characteristics while preventing you from noticing others. Around a third of St Andrew's students are Scottish. They have an incentive to choose Scottish universities because they get free tuition, and as you can imagine, this is particularly appealing for those from less affluent backgrounds. So it certainly won't be the case that everyone you meet is rich. It's also a popular choice among international students, meaning you're likely to meet people from all over the world.
As for the expense, overinflated rents are a major problem in many university towns. I don't think you'd save significant amounts of money if you were studying elsewhere, unfortunately. I'd stick with your offer.