I really can't see the problem with bus drivers. They're remarkably calm and polite considering what they actually have to put up with on a daily basis. There's the assorted idiots, obnoxious customers, children constantly pressing the bell, people dropping food/drink all over the bus, people counting pennies out or spending ages fumbling for a ticket after they've spent the last five minutes standing there waiting for the bus to arrive, that one passenger who always has to talk to the driver instead of just getting on and sitting down, the self-important types moaning to the driver that the bus is late (I'm sure they're aware, and can't do much about it, and your whining is making the bus even later), people wanting the bus to be early for them even if it means missing other peoples' stops, people complaining about prices/services, people who don't know what a return ticket is, tourists using them as an information booth and so on. One person wants the bus to leave on time, while someone else wants it to wait for them to finish running towards it. Can't really win, can you? Add that to the constant stress of driving 20 tonnes of metal around in city traffic all day.
I'd go mad within about an hour of doing it myself. And to be honest, in 20 years of using buses I don't recall any particularly bad experiences that were caused by the driver and not me.