I'll happily give up my seat for an elderly person, I have absolutely no problem with that, my problem is that my uni bus service is quite popular for elderly people due to where it stops between campuses, and I've often gotten on a bus to go to uni and have found it's been full, mainly with elderly people, and as a result of that I've had to miss or be late for my lecture until another bus came along. It's no good trying to get an earlier bus either, I've often qued for the bus hours before I was due to be there.
I know it's more of a problem with the bus service rather than the people, but I just sometimes get a bit angry over it when you think that you're paying to be at university and you can't go to your lectures because of a crappy bus service.
Other than that I don't think them using the bus makes it 'less pleasurable' - frankly I've never thought of any bus journey being pleasurable! It's more the people with loud music and the bad attitudes that tend to annoy me. And sometimes when I'm sat in a window seat and someone sits next to me - not just an old person, anybody - then when I go to get off they act like I've horribly offended them by politely saying "excuse me!"