Hello! First off - it's great that you asked about what they expect to see at an open day, that's a really good head start.
My standard portfolio advice is: always read their particular portfolio guidance, it will often be linked on the course information page under entry requirements. On average, it is between 8-20 (9-10 in your case) "pieces" which you should consider annotated "pages" or slides of work, including development/iteration work for that piece. Include a combination of examples of iteration/development/design work(often the kind of thing you might do in school), some drawings from life (e.g. still life, figure drawing, technical drawing) and imaginative work (work that demonstrates combining ideas imaginatively). Some places might request a variety of mediums, and this is often course-dependent. In the case of Game Art they often want to see evidence of your ability to work in 3D space - be this physical modelling like clay or sculpture or prop-making/craft work, or digital 3D modelling examples.
In regards to your specific questions:
What quality do they expect?
You want to showcase your best work to give you the best chance at a place on the course. You want to be proud of the pieces you're showing off - while still being able to acknowledge where you can improve, if you're asked in an interview. It might be worth looking at examples of Game Art portfolios online, even if they weren't applying to the same university as you.
How much iteration work to include?
In general, they are really interested to see your development work - it's a good idea to include lots of it if you have the ability to - yes even sketches and thumbnails. There will always be pieces where you feel like you have more of this than others - that's normal. I get that sometimes it does feel like you have too much though, and there's no way you can format it all nicely! In those cases you might want to focus on the stuff that felt like it really contributed towards the final design - if you did 8 sketches and 4 of them contributed features towards the final design - maybe just include the 4.
Hope this is helpful to you - good luck with your application!