You mention you both live in/near London - it's a big city, why not pick somewhere other than where either of you live?
I know you think nothing bad would happen if you went to his house OP, but another reason to see him somewhere out and about is that it's neutral territory for you both. You may feel a bit uncomfortable being in his house as all of it - including seeing him in person - would be new and unfamiliar to you, whereas he would get to feel at ease. You may not come across as quite yourself - I'm sure most people would behave differently under those circumstances; not knowing where mugs are if you want tea, not knowing where the bathroom is/where they keep their loo roll etc. It puts you in a position where you're at least somewhat reliant on him and may feel like you're asking permission to find/do stuff.
Better to pick somewhere different but interesting for both of you, where neither of you will automatically have the upper hand.