I think it's more about Entitlement and understanding 'Skilled Worker' status, that most people don't understand about immigration.
A lot of British people feel entitled to a job, welfare, healthcare, education etc. and let's be frank, many of them disrespect this privilege. We are not British by hard earned work, we just here by luck.
Meanwhile, an unskilled worker from abroad might never have had any of these privileges, and given the right opportunities, they could flourish and be an asset to society. You have to remember that it's harder to become a 'skilled worker' if you're from a less developed country. The opportunities are scarce and even if you end up in a high level profession like a Doctor, you'll have to retrain or even be demoted. Either way, closing the door on immigrants might save some money now, but in the long run, it significantly slows human development for the whole world.