Peterhouse was founded by the Bishop of Ely in 1284. Where did the Bishop of Ely get his money from? I am no expert on the bishopric of Ely, but I suspect it came from tithes, fines from church courts and extensive agricultural operations which would have been worked by peasants. Peasants that were indentured to a Lord were effectively slaves. The bishop I expect didn't do very much, he just lived off other people's work on the fields in one way or another. So when he gave money for the foundation of Peterhouse, I'd argue that in a moral sense it wasn't really his to give. Of course, this isn't true for all colleges, but it is for some of the older colleges.
You could argue that cities such as Liverpool were even more directly built on slave labour (in the eighteenth century), but this was no longer a factor by the time their university was founded. I feel that if you are to revel in a long history, then it is wrong to sweep aside any undesirable episodes.
The reason Cambridge is no longer as important as it once was is simple. In the hey days of the great universities, the first half of the twentieth century, top academics wanted to be together to exchange their ideas and so on, so they congregated at a few elite universities. From the 1950s onwards this became no longer necessary. Jumbo jets made meeting for international conferences a lot easier. Fax and cheaper international calls and now email and internet mean that academics can collaborate with eachother only meeting once a year at a conference if they want. So there's no longer a need to group together in just a few centres.
Oxbridge now hopes to be inclusive. Probably they are much more so than the media makes out. But in the past they managed to do their best to forbid many leading figures - mostly non-conformists - from coming there. A leading examples in Birmingham would be Joseph Priestley, James Watt and various other members of the Lunar society. So I would argue that Oxbridge has never really had a monopoly and far from it.