Yes, absolutely definitely worth going.
I've been to three this year and have another next month.
The first was the university I thought best suited for the Masters Degree I want to do. It is a building site, the students are clearly just an inconvenience that has to be brought in to make money, the library was closed on the open day (WTF???) and the two people from the faculty I was interested in only had out-of-date leaflets and they gave up at lunchtime and went home. So very, very, very glad I went.
The second I visited was just out of curiosity because it is local, small and their Masters is cheap. I was SO impressed. Immaculate environment, the library staff and resources excellent and I discovered the faculty has specialist affiliations. The atmosphere was one of quiet, industrious, study. It felt like somewhere I wanted to study. I emailed the head of faculty with some queries and got a lovely personal reply. Had I not visited, I would have not considered it as an option. It is my first choice.
The third has an international reputation in the field I am interested in. The faculty was one of the first in the world and had to fight against the system to stay in existence. But, it came across to me as a graduate factory, taking in students and cranking out clones trained for one specific purpose. I want something for me, to learn what I need to know, not be treated like a school child. This is not the university for me. I was really surprised at how that visit turned out.
The fourth is next month. International reputation, red brick, one of the oldest English language universities in the world. It is totally impractical for me because of the location, I cannot see how any way to make it work for me. But I'm going to compare its teaching with the others I have seen, to determine whether my current first choice is the right decision.
tl;dr: Had I done no visits, I would have made a very wrong decision. You cannot rely on web sites, glossy literature or an old reputation.