Why would you assume college tutors would be less neutral than academics who work at the faculty? Or more likely to get into petty disputes and hold grudges than academics at a centralised faculty? The faculty isn’t an algorithm, if the faculty arranges teaching, this still means that a senior academic is deciding who should teach what and to whom. In the case of Oxford, the senior academics in the Faculty are the same people as the college tutors, just wearing a slightly different hat.
A tutor’s job is to do the best for their students, and this will be overseen by the Senior Tutor at that college who keeps track of who is teaching the students at that college and on what basis. Tutors are senior and professional academic staff and take their responsibilities pretty seriously. Tutors are also contractually required to do a certain amount of teaching - they can’t just say “oh I only taught 2 hours a week this term because I haven’t been willing to speak to anyone who goes to the same college as Professor X teaches in since he slighted me at a conference last year”.
Also, unless you are applying for a tiny subject, Oxford faculties are huge, especially when you add the many talented early career academics, junior research fellowship holders, and other post-docs and post-graduated around. It’s just not plausible that there is only one person who can teach a certain course and if there were, it would be made crystal clear to them that part of their teaching duties would be to teach anyone who wants to take that course. College tutors aren’t a law to themselves, they are still accountable both to their colleges and their faculties.