It's just classic stereotyping really. Some people who voted for Brexit happen to be racist, idiots then exaggerate that fringe and try to link the Brexit "movement" as a whole to racism and xenophobia because it suits their campaign for remain. Then of course there's the people who think that any restrictions on immigration are "racist" even though EU immigration is mostly white Europeans.
As a Brexit voter who hates a lot of what the Conservatives stand for, would never vote for UKIP, is somewhat politically undecided and who did look into both sides, it really ****es me off.