It isn't impossible to get into. When I was at school, a girl in my AVCE class got in there for fine art (through clearing) and her fine art work was pretty mediocre. I'd call to check if they'll take you without the A Level but as others have said, the portfolio is the major thing and if you think you're good enough I'd say go for it. If you get an interview, make sure your portfolio is up to scratch and research graphic design, including designers who have influenced you and make sure you know why you like their work.
If CSM doesn't work out, I'd try London College of Communication (formerly of printing): it has quite a few famous graphic design alumni and may even be better than CSM for the subject.