My gut instinct is to tell you to just do the psychology degree, if that's what interests you - the content will be way more interesting and more readily comprehensible to you than the boring stuff you probably hated in the physical sciences.
In terms of statistics, I'm guessing that most psychologists don't actually like them - they are just a means to an end. Ultimately, all you usually need to do is understand a few basic principles of statistics, which are just logic & common sense really, and memorise some recipes/rules of thumb for deciding what test to use in what type of situation.
Clinical psychology doesn't usually involve any stats on a day-to-day basis, but you'd need to know a bit about them to understand some of the literature. I suspect most people get by with a very rudimentary knowledge of stats.
I hated stats, avoided them at A-level, but working in a field which has a scientific literature has just got me used to using them. They've kind of grown on me.
Good luck & have fun!