Let's pause for a second.
Would there be anything wrong with an African person wanting to meet up with people of the same culture? No.
Would there be anything wrong with a Middle Eastern person wanting to meet up with people of the same culture? No.
Would there be anything wrong with a French person wanting to meet up with people of the same culture? No.
The list is exhaustive. People can get really touchy about someone expressing the desire to meet up with people of the same culture because of the racist connotations (as he's white).
There's really nothing wrong with it. The OP is welcome to seek out his culture. He has racially diverse friends, he isn't saying he won't hang out with non white people, he just (admittedly very inarticulately) wants to spend time with people from the same culture as him, possibly to deal with insecurities he's having with his cultural identity.Most people act as if because people are white, they don't have a right to feel the need to be in touch with their culture and background. That's not the case, it's very important to feel a sense of identity.
Personally, I have never felt this desire. I know two white guys who are my friends, and then at least 10+ in my close circle that aren't. I don't find it to be an issue, as I'm quite secure in my racial identity. Perhaps OP, you should work on that yourself. Try to just accept who you are, and you'll start to realise that there's no real difference between races, and no need for all these feelings.