I ignore them, and move away. Punching them in the face is just giving them a reaction. From what I've seen, most guys won't just walk away if you retaliate, whereas if you ignore them they get bored.
I couldn't comment on whether I mind or not though, I've never actually had my bum pinched or anything "rougher" like that. The most I've had is an arm around the waist or someone touching my hair, in which case I just move away. I'd be pissed off if someone actually slapped or pinched my bum though, although I wouldn't resort to punching them.