I am with the majority here on my ranking of them as maths departments. Southampton first, it has a serious mathematics department. (about 7th largest). Lively and research active, lots of postgrads, big enough to have wide coverage of maths teaching for undergrads. The applied people there for example very active in mathematics with industry for example. Sheffield next, definitely on size and quality between Southampton and the rest.
The other three are smaller operations and you have to decide if that is what you want. As I have said before a smaller department really suits some people but you will usually sacrifice coverage of mathematical topics especially in third year courses. Also if you visit see to what extent you think maths has a separate identity as a department. Sometimes if maths is taught in a small corner of a School of this That and the Other (eg Exeter it is in a "School of Engineering, Computing & Mathematics" ), and it is not clear that you will pick up what it actually means to be a mathematician. Also lack of a separate identity for maths in a larger school makes it vulnerable, especially if the other parts of the school are better at attracting money and students.
As for the building yes it matters, a bit at least. A new or well maintained maths building reflects a university prepared to invest in maths, helps attract and retain good lecturers and generally raises moral of both students and staff. Certainly something we observed with the new maths building in Manchester. But it is neither necessary nor sufficient for a good maths department!