It's hard, but you don't need to be a genius; you can do well with hard work. Different people do better than others of course though, it depends on your learning styles and strengths.
The difficulty doesn't necessarily increase with entry grades; entry grades reflect how competitive a course is more than how hard it is, having said that, I haven't taken exams/modules from all the unis in the country. I did my first year at York, then moved to Liverpool and the standards seem to be the same in terms of difficulty, but they have very similar entry requirements, and aren't exactly that different in league tabling etc...(not that they mean anything anyway). However, when revising for my second year astro exam (which is an optional module for third year straight physics students), I came across a previous exam paper from Oxford's equivalent third year optional module, and it was basically the same in terms of content and difficulty. I know that Liverpool/York to Oxford again isn't exactly the biggest jump in the world, but if there's no change there, why would there be a change a bit lower down? Generally physics degrees have at least ABB or AAB requirements, just because it is difficult though, so you don't get so much of a range of offers as some other courses may get (although some are more lenient than the others).