The colleges are individual institutions which are members of the university but are independent in theory otherwise. Undergraduate admissions are handled solely by the colleges, while graduate admissions are handled centrally by the university and department(s) involved. Pastoral support tends to be dealt with by colleges individually, as will all accommodation matters for undergrads (and possibly postgrads as well), although there will (probably) be a central accessibility/wellbeing department/initiative outside the college system. Also for reference, each college is not a single building (usually). They will normally be a collection of buildings, usually on a single (closed) site, although some might have "satellite" sites with accommodation for e.g. later year undergrad students or postgrads.
Teaching departments are separate to the colleges, although colleges arrange teaching internally for undergraduates (which can make up a substantial amount of the teaching for some subjects, notably "arts" subjects such as e.g. history, English etc). However, even for subjects where much of the teaching is done via tutorials in the colleges, you may not necessarily have tutorials in your own college, particularly for specialised options in later years of the course (where the director of studies will arrange the tutorials with whomever has appropriate expertise in that area. That said, I gather this is more common at Cambridge than Oxford
That said, there are often classes, lectures, and labs arranged centrally by the department outside of the colleges for many courses though e.g. any course with language elements will probably have centrally arranged language classes, some "arts" courses with more technical content like economics are likely to have centrally arranged classes and lectures which make up a major part of the teaching Additionally any STEM subject will have centrally arranged labs and the lectures offered by the department for those subjects are usually more core to the teaching for those subjects, and very small subjects may have more centrally arranged tutorials and classes.
However for Oxford (and Cambridge), due to how the tutorial(/supervision) system works there, which is done via the colleges, the colleges are more than just a named building or student accommodation. For other collegiate universities (e.g. Durham, York) though they are, as I'm told, much more just glorified halls of residence.