It's only a 1-hour lecture, usually, so you're generally expected to have the courtesy to go beforehand/afterwards.
They don't try and stop you or anything, but it looks a bit rude, and is distracting for others.
In tutorials with 5-6 people, you would have to ask your lecturer for permission.