as a libertarian I've thought about this subject for a long time. I *can* actually find cases to "limit" (not necessarily ban) speech.
1) threats of violence - this is 100% clear
2) breaking legal contracts, i.e. promising by law not to say something in exchange for the consideration for you. this would include copyright, branding products with false information/ingredients listed and similar cases. the first example (copyright) is about property rights, but things like deception to customers is more to do with the similarity of it with theft - if you say "I'm selling 2 dozen eggs" and it's actually only 3 eggs, and only the seller is in the position to really know, then that's obviously theft, because they're making an agreement that isn't based on the correct exchange and more towards giving the seller too much for what is objectively not what is being sold.
3) nuisance - if you're using speech to make incredibly loud noises right next to another person's property, robbing them of the value or benefit of their property, then that is akin to economic sabotage, and hence, is a kind of property damage
4) psychological damage - this one I've wondered about for a while - cases where people become so distressed by other people's language that they sustain medical injuries (psychological, and that's to do with the physical brain). if there is overwhelming evidence of this being the case (and that's extremely hard to even imagine), then I suppose then the person responsible for the damage would need to pay damages, but the speech wouldn't be banned because you have no idea who's going to be ridiculously thin skinned. this is in line with the non-aggression principle, technically. I'm not sure if this should include truthful statements though - freedom of speech is here partly to extend the truth, so if something is disturbing and damaging, that doesn't necessarily mean that a speaker of truth should be punished. it's only if it's purely psychologically abuse, and emotionally abuse. if a fact happens to be as such, then that's a blameless case - if I say "god doesn't exist because x" and a christian or muslim freaks out and starts cutting themselves, they aren't doing that because of me, they're doing that because of the facts that nobody can change (clearly I'm talking about the scientific and philosophical challenges against god)
5) libel/slander - i.e. spreading lies and causing a person to lose their property (wealth) due to it. I'm very conflicted on this question because it is essentially a case where one person says something (lying) and others happen to believe it - you might ask "the truth will prevail - why doesn't the victim just explain the truth to beat the liar?" - well, that could be extremely expensive. if the thing the malicious person said was actually true, then I don't think that should even be illegal, but if it *was* a lie and the victim had to spend a lot of money in campaigns and adverts to prove that the lie was as such (i.e. so they could get back everything they worked for in their life prior to the extensively damaging lie), then the malcious person should pay damages for that, but the actual speech itself shouldn't be illegal or banned because, like I've said, we can't know for sure what people will be willing to believe, and we have no way to know what lies have such serious consequences
6) acts to necessarily facilitate criminal acts - i.e. if a person says something deliberately to cause people to then act criminally, and they wouldn't have been able to get away with the act without the speech, then that ought not to be allowed. this is different from "encouragement" which I don't think should be illegal. I'm talking about enabling cases, i.e. if a religious leader, respected by everybody, had very easily mislead followers, and he knew of this fact about them, and abused his authority, knowing that it would have criminal consequences. this is different from saying something like "I think *politician x" should die" in a non-serious context, even if a lunatic happens to be very easily swayed by that speech and kills them.