There are two possible outcomes here:
1- You go and see him, and although your pride might be a little dented, you'll talk and sort it out. Find out what hes been saying, what hes heard about you and set the record straight. If he believes mindless gossip from the street about you and believes it, is he really worth it?
2- You let pride get the better off you, and leave it. Who knows what would happen after this; he might come running, he might end it, who knows but its a bit of a risky option.
Even if you decide you no longer want to be with him, wouldn't you rather end things personally and on friendly terms? Personally i'd go and see him and try and sort things out. Sorry i've not been much help here!