Its always hurtful when people gossip about us - we cant help that we find it embarassing and shameful to hear things said about us that arent true, and we just want to set the reccord straight. Unfortunately this isnt always possible and the more we protest the more we appear to have guilty consciences causing us to protest too much. If you make a big deal out of something, people assume there is some truth in it. Its possible that the fact you used to fancy this guy is making you hyper sensitive and extra defensive, because the accusation is too close to the truth even though it isnt accurate anymore.
The best thing you can do is relax, laugh this off, and act as if the gossip isnt worth your bother. If you went to a party with someone like a brother or a father and people spread rumours that you were together you'd just laugh it off and forget about it, because you knew it was 100% false, and thats the only thing you can do to convince the gossips that there is no truth in their imaginings.
People will always love to gossip, Im afraid its what most of them do. You cant stop them doing it, but you can change your reaction to it.