There's not loads of maths in Pharmacy - it's important, but not as important as chemistry.
I won't lie and say that studying pharmacy is easy, because it's not. I went into it thinking that it was (afterall, my dad did it so how hard can it be - although a lot's changed since 1974 it turns out...).
IME, the first year is definitely the worst - after that it kind of eases up, but in terms of grades and make-up of your final degree classification, the years get more important as you go.