Things like libel, false advertisement, perjury, making credible threats, and so on, would be reasonable limits on free speech.
However, 'hate speech' and causing offense are too subjective as standards and can be easily abused in order to limit expression and to silence unwanted opinions. To merely 'offend' someone should not be a crime. Would you rather live in a world where people learn to deal with their issues maturely and independently, or one where people run to the authorities like petulent children whenever they hear something they find provocative?
What does offense even do, anyways? 'Someone drew a mean picture of the prophet!' Well, so what? Was he hurt?