Why do you like him?
Are you usually shy and quiet? Maybe he just doesn't vibe with those types of people. You should just ask him, I did that before with a stripper girl in my class that assumed I didn't like her we talked it out and it was just a big misunderstanding. How do you know he hates brown girls?