I'm not too short sighted, and I don't wear them much in public (wear contacts). I don't even need them either, because my sight is only bad when it comes to reading signs, trying to figure out what the bus is down the road, reading the black board etc... that kinda stuff.
So yeah, I would definately take them off.