I think it's polite to drive more smoothly with passengers - take corners more slowly, more care to avoid bumps and potholes, make the ride smoother in general and reduce acceleration/deceleration. As a driver, you're constantly holding the wheel in front of you, and you can anticipate every move, as you're causing them, so it's easy to feel stable. As a front seat passenger, you have nothing holding you in place other than the back of your seat, so are likely to get thrown around a bit more by a rough ride.