I’m in the same boat! Next time they make such complaint, just tell them that you’re always open for a conversation and they’re welcome to initiate conversations. For some people, socialisation is a gradual process and forcing it only makes for a really unpleasant impression. Encourage them to make first moves, thereby widening the opportunity to know them better. Only use this tactic to whomever makes such comment.