I would say try it and see how it goes. But if you really feel you know it is not for you, try to change to a preferred subject, perhaps at the same uni. The inflexibility of the system is a liability, very unlike undergrad in the US, where you can experiment and it's almost expected you will change your path.
My daughter had a friend in physics at Cam and she dropped out in the first term and went to Manchester to do literature, which she loves. There is no question she did the right thing, but she did try Cam first, however briefly. My daughter was fortunate enough to know what she wanted and happy to stay in it.