If that male goes to the police and says I'm harassing him (because others maybe asking him and are saying stuff to him about what I said about him, his behaviour)
What if the other people keep pestering that person?
Unless you've set these people on him (like a pack of wolves), then they're responsible for their own actions. They're all grown adults, presumably know right from wrong, so they're responsible for their own behaviour; even though you may have created the situation.
Even if you did put them up to this, he would have to be able to prove that you encouraged them
I would tell them to stop. But if you have no connection to those other people who are doing the harassment and have not asked them to give the guy a hard time - ignore it - completely