I'm re-reading them all currently once again but I've come to the conclusion it's The Order of the Phoenix, the first 4 are basically set completely in the school with the exception of the World Cup and it's only from The Order of the Phoenix that Rowling's World opens up completely beyond Hogwarts and you begin to see how much is actually going on. Plus, as mention a number of characters suddenly become a lot more prominent, such as Luna (although the Lovegoods do get a mention in Goblet it's nothing beyond a name) and Ginny becomes a main character too. It also, I'd say, blurs a lot of the lines between the 'good' and 'evil' characters when you see how characters in good roles can so horribly abuse their positions. Just my thoughts but it was actually my least favourite the first time I'd read them all but it's become my favourite over the years just through the level of detail and breadth it adds to the world.
/thoughts