First. he's probably not being serious. Second, I don't know if it's the case for all but I assume that most halls require some kind of key or pass to access the flats for security reasons. You can literally not let him in if you don't want to or say that you're in town and just wait in your room until he leaves. As others have said, pretty sure he's not being serious as it's probably a long way to get to and from your uni every other week or so (unless you go to a local uni) so it would cost him a lot of money in travel costs alone.