I can't sleep when I'm sharing a bed with someone. I don't know why, it isn't a space issue or anything, I just can't do it. If I had to be up early the next day for work, I probably wouldn't stay at a boyfriend's house either.
Maybe he just appreciates his own space, that's the vibe I got from what you said.
As others have said, some people do just like their own space when they sleep. They like to wake up in their own home, with their own plan for the day and thats it.
I'm not one of those people personally, but I must admit I get a far better night's sleep when my boyfriend isn't in the bed too rolling around and stretching etc.
But its obviously upsetting you, so talk to him about it, ask him why he's so reluctant to stay. He probably doesn't realise how much is bothers you.