Using a racist word or term does not necessarily make someone 'racist'. My OH is Indian (British-Indian: her parents come from there but she was born in the UK, has a generic UK accent for her city and doesn't act/dress according to her heritage) and we have the kind of relationship where we use silly faux-insulting words for each other that we would never dream of using with anyone else or if anybody else was nearby, always on the understanding we never actually mean the insults. For example if I'm off out to Tesco she might text me "Oi tw*t don't forget the milk" and I may reply "Already got it, sh*t-for-brains".
If we're having a mock-argument she might say "You're talking bo**ocks you stupid white w****r" to which I might say something like "Go away, p*ki-slag" (Yes I know the difference between Pakistani and Indian....).
Again, we would never speak like that if anyone was within ear-shot and neither would we use those words against someone else.
I don't know the guy personally of course but Cumberbatch is not racist. For the record I don't consider the word 'coloured' (in that context) to be racist anyway. Outdated, unwise and inappropriate yes but not racist, and neither do I think he was being racist at the time.
My bugbear with this whole incident is how grovelling his reply was. By all means apologise if anyone was offended by his choice of words but he seems to be falling over himself in his response. And this applies to anyone else who has had to publicly apologise for anything deemed racist. My memory isn't the best but I don't remember any other situation involving offence (ie sexism, ageism etc) where such emphatic denials/apologies are always required.
Nobody in their right mind would suggest racism should be treated lightly but in this day and age I'm uncomfortable with this perceived 'guilty until proven innocent' attitude surrounding the subject.