I know it's hard no to take prejudice personally, but it's a natural precaution all human's take.
When me and my friends are hanging outside smoking trees I appreciate and accept we probably look intimidating to some, I can't take this personally though, we would never hurt a fly, but unfortunately there are people out there who would, and phrases like "better safe than sorry" exist for a reason.
I'm wouldn't say black people scare me because that would be ridiculous. But if I was walking alone at night and there was a black guy walking towards me, I won't deny I would be scared. Not because he's black, but because it's dark and I'm alone. For all I know he could be scared of me (not that I could scare anyone if I tried).
Could this happen to you?