I’m too scared to further our relationship

I had the same fear the first time I met up with a guy. I took precautionary methods and texted a friend my location, updating them every so often. I also made sure to, throughout the date, not stray too far from crowded places. You can take that route, or trust your gut on this one and tell him you don't want to go (I say this because both options allow you to feel safe). You are allowed to change your mind about going to the cafe - if he is serious about you, he will prioritise your comfort. Take this opportunity to be open with him about the fact that you feel like things are going too fast. Moreover, wait until you trust him to have sex. It will be a significantly less harrowing experience if you are comfortable with your partner. In the meantime, I would advise you deal with your distrust of men. The most simple way is to take baby steps with this guy. Open yourself up to him as he does, making sure you both know that each other's comfort is very important every step of the way. I say this because if you can learn to be genuinely and thoroughly comfortable with this one guy, it may help you with interactions with other men as you will learn how to navigate the relationship. Soon, it will feel right to meet up with him and/or have sex rather than scary.

(Make sure you genuinely like this guy for who he is, and that you're not just pursuing the relationship because you want companionship.)