He was giving you a **** Test. You failed this particular test. There's 3 main ways to pass them:
1 ignore what they said and just carry on saying whatever you want to say and talking about whatever you want to talk about.
2 misinterpret. And do it in a way that you pretend he was complimenting you. EG "Aw that's so sweet, you've been talking about me with others..."
3 agree and exaggerate. EG "Yes, I agree! I'm so much of a .... that ... (some stupid ridiculous crazy exaggeration)"
The root of all this is frame control. Your frame should be that you are a cool and attractive and worthy person. And nobody's gonna knock you off that frame.
When you fell into the frame of the stupid malicious gossips he'd been listening to, it did you no good. In future don't let your self esteem be undermined like that. Deploy your cast iron emotional strength and maintain your frame.
Go ahead and talk to him next time you see him. And do it in a way as if you never had the previous negative conversation.
Don't qualify nor justify yourself to anybody. Especially people you are attracted to. And watch out for people putting you in a position where they are expecting you to qualify or justify yourself.
Most of the time, when people criticise you it's more a reflection of something not right with them than it is about something not right with you.