I've seen this a lot and I started having some of the same issues, so I'll try to give my perspective as an Oxford offer holder as well.
Your situation may be entirely different, but chances are if you got into Oxford, you've worked hard. But if you are worried about losing interest in your subject and the future workload, you feel as if Oxford will be a huge step up from the amount of work that you had to do to get in.
What I can say then, is that how hard you have worked already to most students is actually far harder than you probably think in your own eyes, and this is true for almost any high-flying student. But you've become so normalised to it, you (hopefully) have still been able to have fun, relax and not completely stress yourself out. Being at Oxford will be the same principle. It is a lot of work and it will be a step up, but it's work everyone there does, your mates will have the workload and most students grow used to it to the point they're still able to have tons of fun, get involved in clubs and societies, and don't spend everyday wishing they were at a less intensive university. After all, if you start at Oxford you won't have experienced anything else.
No matter the university you end up at, the grass will always seem greener on the other side. But the advantage of an Oxford degree over other degrees (depending on the course but generally) is pretty certain, not just in terms of prestige but in terms of the quality of teaching and opportunities available. You've discovered yourself in your gap year, but no one can say whether that will change or not. Few people don't change from 18 to 28 let alone 18 to 48. But the benefits from a marginally better education will likely still persist. Bear in mind, at other universities you'll still be working hard, managing your time, stressing out and nowhere you'll go will be super straightforward, easy or tons of fun if you want to end up with a good degree. If that extra enjoyment you may get definitely appears worth the sacrifice from Oxford, then your mind is already made up.
If you don't think you are good enough, admissions tutors disagree. Absolute worst case scenario you'll be in a great position to switch, but doubt is normal. Support at Oxford is very good for these sorts of things because it has to be. Oxford will push you more, but if you can manage it you'll come out better for it, and personally doubt isn't ever worth the sacrifice of turning it down. The workload is greater to answer that question, but it's not impossibly harder than any other university as mentioned. Everywhere will be tough at times, but everywhere you'll be normalised to it.
I've given a pretty one-sided view, which to be fair is my own, so I will add that an Oxford degree isn't everything. For some courses like Medicine or other vocational courses, going to any top Russell Group university will have you set. For most courses, the benefit stops are slight prestige which is becoming increasingly less important. The soft skills gained from an Oxbridge education through work can easily be gained at any other university, and if you genuinely think you will regret your time at Oxford, I think it's an entirely respectable view to turn it down for somewhere you'll feel more fulfilled. In the end if you don't think you will regret turning down Oxford, given what has been said, then go for it, your happiness matters a lot more than prestige or the slight benefit at the risk of regret throughout and afterwards. Just in my opinion, if you are unsure, you have nothing to lose by at least trying.