It really depends on what your subject is. If you’re doing a humanities course, the only issue is if you have a tutorial or class which would clash. In that case it might still be possible to get around though, especially if there are multiple groups (which I guess is pretty likely if you’re taking a compulsory first year paper) and you could just swap into another which didn’t clash; in my experience, most tutors are perfectly happy for students to swap among themselves like that so long as you shoot them a quick email to let them know beforehand, and you don’t have to provide an excuse or anything. If you can’t swap groups, you could also ask the tutor and your tutorial partners if they’re willing to move the tutorial to another time — this is perhaps a bit more risky and will depend on the tutor’s availability etc.
Lectures tend not to be compulsory in any sense, and you can always catch up on them because at the very least the slides/notes (and sometimes a recording) should be posted to Canvas. If it turns out you don’t have any academic commitments on the day in question then you’re free to come and go as you please — it’s a university, not a prison (or worse, a boarding school…)! In general, the kind of thing you’re describing is not uncommon at all — subject to the above constraints, there definitely are students who will leave Oxford during the week to visit family or go to events in London.
As I said, this more applies to humanities courses, where you will generally only have a few compulsory contact hours a week. If you’re doing a science then labs may present more of a difficulty in getting out of if they clash.