Theoretically, if it's in a building that doesn't require id at a turnstile, any non-student could just walk into a lecture of any university and nobody would say anything
You can walk into any of my lectures especially as they are 150+ people and the lecturers don't recognise anybody anyway.
In addition to what others have said, it's often possible to go to public lectures - you can find them by googling, and they are meant to be interesting and understandable by people with no subject specific knowledge.
Bristol Festival of Ideas is a very good series of such public lectures.