I think it really depends on whether you'll enjoy the experience as a whole at Oxbridge. I hear all the time that coming to Oxbridge is essentially giving up a good social scene for the sake of (what is perceived as) a better education.
But personally I think the Oxbridge social scene is fantastic and I love Oxford way beyond the course and its intellectual rigour. Personally, I don't believe I would enjoy Leeds as much socially or academically. My point, therefore, is that you should not rule out Oxbridge's social scene without considering whether you would enjoy it.
The social scene in Oxbridge as opposed to somewhere like Leeds, as far as I'm aware, is that it is a lot more intense. Since we have to work harder, we tend to play harder too. I often find myself busy with back to back events all day every day for a week. That said, like the workload, it's more than manageable. I'd say the night life is better at Oxford than Cambridge, but probably nowhere near as popular as at Leeds - but it's still definitely possible to go out 2-3 times a week if you want. A lot of the social scene also revolves around extra curricular and perhaps a more eclectic set of activities - people are interested in anything and everything, which means a lot of people spend time on, say, classical music, or whatever they're passionate about. For me that makes the social scene far more enjoyable.
If you think that would suit you, I recommend giving Oxbridge a visit (there's an Oxford open day in 2 days btw). To be honest I recommend giving it a visit anyway. It's better to have ruled out Oxbridge definitively than to always be thinking "what if".
But I strongly recommend against applying just because other people think you should. If you wouldn't like the course, the intellectual rigour or the intense social sphere, don't apply.
I have many friends who were dead set on Cambridge because of its reputation who, since getting in, hate it. I also have a friend who loved Leeds but applied to Cambridge too - he got into and now loves studying at Cambridge, and regrets nothing.