We're all African if you trace it back far enough. Having said that, it's not difficult to see why anyone would be offended for being mentally categorised like that. Not all people of darker skin are African, not all Oriental-looking people are from the Far East and not all Caucasian people are from the Nordic countries or the Swiss Alps. What about people of more than one ethnicity? Educate yourself and think before you speak.
p.s. This is definitely a case of sheer ignorance, though, and not racism. Your colleague was inappropriate to brand you a racist. There's a difference between being racist and saying something silly or offensive.