Why are Brexiter's associated with ring wing and racism?

I am left wing, very liberal and encouraging of equality and diversity, and don’t consider myself racist or prejudice in any way. And I voted for Brexit. Why do people still call me out as right wing and racist for this? Brexit wasn’t a normal political vote, it was free from parties, therefore surely free from ‘wings’ and party opinions? E.g. Boris and David both in the conservatives but different views on the referendum. Also why are Brexiter’s being associated with people who voted for Trump?? (I am defiantly not a supporter)

A lot of the things being promised by the Brexit campaign seemed hopeful. Okay I’m not sure if we made the right decision in the end but the fact we were even having a referendum made me think it was needed. Surely we wouldn’t have been given the choice had it not been a good choice to make? The immigration side (which I assume is where the ‘racist’ thing comes from) wasn’t a deal breaker for me at all, but at the time it seemed to be the peak of the Calais crisis and yes I was worrying how we’d handle it over here. Not because I’m cold hearted but because we weren’t prepared enough to help it, and it would have probably caused chaos.

Just curious for some opinions