You can't see the fit of anything under a gown. It would look a bit odd not to be wearing a jacket, so as long as you can move in it, I'd wear it. The gown covers shoulders, waist, hem line etc - it also drags things out of shape anyway.
Generally though, attendees at formal events at Cambridge are so international, that while it might seem 'the right thing to do' to fit in with the formality, in truth many people don't and come in their own interpretation, and that is fine. Why on earth should everyone perfect an upper-middle class, English, 1950's style of dress?
With any luck, someone will come in a kilt, and that will take all the attention!