I don't think it's in any way moral to ban images, no matter how offensive.
I can't see a truly justifiable reason for any flag to be banned. And it'd certainly be strange to ban the Confederate flag in Britain when we don't even ban the Nazi flag, which is a far more relevant symbol of racism in Britain.
But, no, I would consider it to be a gross violation of freedom of speech and freedom of belief. The state shouldn't encroach upon matters where direct harm is not caused upon a non-consenting individual. For the same reason I strongly object to the worryingly regular occurence of the police getting involved whenever someone tweets or posts on Facebook about something interpreted as "racist". It's not right to criminalise belief, and no matter how morally reprehensible we believe it to be, I don't think there is anything legally wrong with someone privately or even publicly harbouring racist views that do not infringe upon the rights of others.