To be honest, we only develop the attitudes that we're brought up to. When today's pensioners were born, there would have been very, very few people who weren't white in this country, and everyone is scared (or at least wary) of the unknown. Attitudes are changing, but only as education changes. Within a couple of generations things will be very different.
It's not like they're being intentionally racist with the intent to hurt somebody/their feelings.
When I was younger I was terrified of being on a plane with a Muslim. I was physically scared getting on a flight to New York in 2006, and even then I was old enough to know better! Yes, it sounds ridiculous now, but after listening to all the media attention, plus opinions of older relatives, I was convinced that we wouldn't complete the journey. But now, through education, I realise that it was all rubbish. But people who haven't had that education will still hold the same beliefs.