are you joking? You do know that the courses at Cambridge aren't run by the colleges, right? As in, regardless if you got into Pembroke, Jesus, Trinity, or St Edmund's, you'll still be studying on the same English course...
Also, to clear this up, St Edmund's isn't a "bad college" or "easy to get into" - in fact, it could be argued that it's quite difficult considering that they only admit between 50-60 undergraduates each October compared to the 100-300 the other colleges let in. Plus, purely as a college, you've got a huge community of international students, it's not *too far* from the town, and, from what I've heard, they throw some of the best formals and balls (the May Ball there has a whole theme park in the garden!)
Oh and the fact you've brought the Tompkins Table into this is wild; as mentioned above, most colleges let in double or triple the amount of UGs so of course the number of Firsts achieved will be skewed by that. I don't have much of a rebuttal for the lack of famous alumni other than it's a relatively new college (having achieved its status in the 80s or 90s or something) and, I guess, maybe the alumni shouldn't be as important to you as the quality of the course.
What I'm trying to say is that I don't think you really seem to be aware of what the collegiate is or how it works. On top of this, unless you're on Wikipedia, your CV/LinkedIn will just say "English BA, University of Cambridge" - 99.999% of places you apply to will not care what college you went to.
Look, obviously UCL, Warwick, and Durham are exceptional, exceptional uni's and, of course, reject your Cambridge offer for one of them if you'd feel more comfortable but, at the end of the day, Cambridge is Cambridge and there's not much more I can say.