I wouldn't say it was negligible for anything but the biggest subjects at the biggest colleges. If you're applying for NatSci at Trinity or John's or something, then that's absolutely fine (though I'd question your motives for wanting to stay together), but for the majority of subject/college combinations I wouldn't advise it because the number of offers will be limited.
How would you feel if one of you got the offer and the other didn't, knowing that all other things were equal? What about after the interview/test, when you'll inevitably be having discussions on how well you think you answered each question?
I personally think it's better to avoid any problems by applying to different colleges. There are enough to choose from, and it's not as if you'll lose contact with each other if you're not living across the court from them. Tbh I think it'll enrich your experience, both the application and (should you get in) your lives here if you apply to different colleges.
Just my advice though, and of course the choice is up to you. I just see it as a worry which you don't need to have.