Not the kind of thing I'd normally discuss in polite company, but since you did bring it up...
Assuming you're male and all you did was urinate. If you don't touch the bathroom door handle with your hands (maybe it has one of those new foot-pedal openers), and manage to aim without... well, touching yourself... then really, there's nothing wrong with it. If you're a girl or you produced any feces, though... then you need to wipe with toilet paper, and that means your hands are dirty and you need to wash them.
I wouldn't talk about not washing them, though. It produces all kinds of gross mental images and violates people's sense of good hygiene unless you go into graphic detail about exactly what you did in the bathroom, which is also unpleasant.
But if you're in a major hurry, and you really need to urinate... then you could just carefully pull down your pants, go, and then pull them back up without touching anything, and run out the door, making sure not to touch the door handle/knob with your bare hands on either entry or exit.
Two things that make this a lot easier in some modern bathrooms where I live, are bathrooms that either lack doors completely or have foot-pedals, and toilets that flush automatically on a sensor so that you don't have to touch the handle.
Even failing that, using a handkerchief or any handy cloth to touch the doors/handles and then putting it away is the next best thing.