It happens, and it always will, you're just going to have to get used to it. God knows how much racist abuse I've gotten over the years. It's just down to ignorance, the best thing to do is ignore them and move on.
Whenever I get it now, I either go along with it and like go WAHEY! or "Excuse me mate, we speak English here, not moonspeak" which they usually find somewhat embarrassing since they aren't getting the reaction they want, ignore them and carry on with my life, or in one case when I was in a bad mood, threatened to beat the **** out of this guy, which resulted in the fella doing one.
The thing is, it usually happens when you're either by yourself or with another ethnic. If you're in a group larger than two, it'll never happen.