The truth of the matter is that leaving the EU gives the UK more freedom- and that freedom can be used for one side or the other.
If you want as many nonwhites as possible to move to the UK, for more "diversity", leaving the EU will give you a system by which you can shut down European immigration.
If you want as few immigrants as possible to move to the UK (white or not), leaving the EU gives you the opportunity to reduce the number of migrants altogether.
Leaving simply means more freedom and that is why many on the left support leave. It doesn't mean a more left wing or right wing government per se. It simply means more power to Westminster and less to Brussels.