Alright, let's just unpack something first, and that is what "mild learning difficulties" can actually mean. As a term it is extremely broad and can mean issues with cognitive ability (which is the closest thing to what you're referring to as IQ), auditory processing (how quickly or effectively you process information that you hear), verbal output (how quickly and effectively you can put together and say things you want to communicate), sensory processing (how effectively you process external elements like noise, light, smells and so on, which can in turn impact on how you can engage with learning), and a whole host of other things. People with mild learning difficulties may need input from speech and language therapists, occupational therapists and so on, or they may not. They may require other provision within school to assist them, or they may not. They will very rarely need to go to a special school; almost always those with mild learning difficulties can be taught effectively in mainstream education. But not only are these things so varied, it's also doesn't mean that anything is actually wrong with you. We all work in different ways, and everyone has varying issues with each of those things I've listed above. It doesn't mean that any specific person has a problem or is flawed in some way. It's just how people are. The only reason the term "learning difficulties" is used is to identify those people (and usually children/young people) who require specific support and provision in their education. Which again, doesn't mean there's anything wrong with them. It just means they need help with some aspect of their learning and/or lives, and we will all eventually need that from time to time.
The reason I mention all of that is for two reasons. First, the guy who is saying this almost certainly has not a single clue what is talking about. Even if he is right (and I have some considerable doubt that he is; being slightly slower on the uptake or being forgetful does not mean you have learning difficulties), that is not an issue in any way. Second, in any event the way he is using this term is at the very least insensitive, and at worst positively hurtful. It says far more about him than it does about you. It seems to be unintentional and not malicious, but it is still not something that he should be doing. I don't think you have anything to worry about personally here because of what I've already said, and I don't think you should be bothered at all by this. So by all means go along with it if you want and treat it as a joke. But if you'd just prefer him not to say it, I'd just ask him to stop. There's no need to retaliate in some way by highlighting his own personal failings. Just ask him not to refer to you in that way, and I suspect that will have the desired effect.