I don't know what it is, but school careers advisors always seem so negative! I'm sure they mean well and are trying to help you be realistic in life. And to be fair, clinical psychology is definitely difficult to get into (like many other job fields) but, if this is really what you want to do, definitely don't let any Debbie downers put you off. If you study a psychology degree, it's quite a 'transferable' subject that, even if you decide that clinical psychology isn't right for you during the course of your degree, it will equip you with many effective skills sought after by employers.
If you give psychology a go, you do most likely have the option of changing courses later down the line. Your teachers are right in a sense, you need English-based skills to study such a demanding subject that involves much writing, but I also think this is true for many degree subjects. It's very important to bear in mind that many people 'peak' when they are at university, outperforming people that got higher grades then them at A Level. I know this for a fact as my mum is a university lecturer who has found this to be true on many occasions. The depth of study really lets you explore your interests. Don't be put off by your A Level grades now.
If you really think you'd like to pursue cog psych, I would say, by all means, go for it. Lots of people that sore successful today would not have got where they are today if they didn't believe in themselves or let other people who (wrongly) didn't believe in them hold them back.