By a small margin, I would think the GWR from Paddington would be your fastest option as the train journey itself is about an hour long.
Though that said, you’d then have to either get a bus, taxi or walk to the colleges that you’d want to look around most. To give you an idea of how long of a walk it would be, Google maps says St John’s College (one of the most central colleges in Oxford) is about a 14 minute walk from the train station.
Driving there should take circa 1 hr 20 m in all, according to Google maps. However, I suspect that this is a rather optimistic estimate and because I’m looking at it at like 4 am so traffic probably isn’t nearly as bad as it would be during more sensible hours of the day lol.
Looking on BusBud.com, it would appear coach journeys from London to Oxford usually are around 1 hr 30 min each way, but they all seem to start near Buckingham palace and drop you off at Gloucester green (which conveniently is in central Oxford and is about a 5 minute walk from St John’s).