I get that alot, also I wear a headscarf so I suppose it makes me look more...different? I live in a very white area so I guess they're just not used to it.
In regards to your question, it really depends. If you were to simply ask what my ethnicity was, I wouldn't be offended (also,try not to make it the first thing you ask) but some people can get annoying.
I remember having a conversation with a lady once and mid way she asks me 'so were you educated in this country?' (I don't even have a foreign accent and except for my scarf I dress very western) When I told her I was she asked when me and my family emigrated to this country. I ended up explaining that I was born in England and have lived here my whole life.
Some people assume that just because you're not white you can't be british and it makes you feel like a bit of an outsider. Like when I said how cold it was and a co worker said 'yeh i bet it's not like that where you're from, you'll get used to it!' .....i've never even been abroad before.
I'd find it rude if someone asked 'where are you from' (in terms of country not city) because it implies that they automatically think it can't be from England because i'm brown.
A better way would be to ask, like i've said, what their ethnicity is. It's also more accurate.