I've done this a lot. The worst thing that ever happened was him giving me his cold, which he apologised for. When we were discussing this, (we'd known each other for several years at this point) he said we need to meet in public. We did and I got a tad lost.
If you're both sensible, it's fine. It's not as bad as people make out. We had planned to meet near a park on a Sunday afternoon. We (because I have no sense of direction...) ended up meeting near my uni and went for a walk, which was nice.
We met up several more times after that. We don't meet up any more due to life stuff. (I'm now 200 odd miles away from him, although, I do make the odd trip there now and again)
Put yourself through the test