"ginger" is a strange one, because unlike other things I mentioned above like calling people gay or referencing their ethnicity, "ginger" is a completely stand-alone and foundationless insult (if you can call it that). When one aggressively calls someone a paki or a ******, say, the offense is carried in the implication that the target shares the undesirable characteristics (real or otherwise) associated with those people.
But, what is meant to be wrong with ginger people? AFAIK there's nothing, not even jokingly - unless someone can explain it? So it's just a fairly random thing to pick up on that became self-sustaining.
That ginger people get upset about being called ginger makes as little sense as people calling them ginger in the first place.