If you have a few in your garden, nobody would even notice. Obviously if you had loads that would be a different issue, especially if you were supplying them to people.
Besides, I don't think magic mushrooms really would grow in an ordinary garden, most grow in more wooded areas as far as I know.