Should I end it with this current friends with benefits?

In early september I met these two guys on a night out, I stayed at their house (in the bedroom with one of them) we didn't have sex/do anything apart from kiss. And I said no to sex. The thing is when I first met the two guys, I really liked the look of the friend.. not the one I stayed with so much.

Before I left the next day, the friend followed me to the door and gave me a hug saying he might not see me again. I was gutted when I left, so I had a look on facebook and found him.. decided just to be straight out and tell him I found him attractive. He replied saying thank you and that he was up for sex if I wanted.

I met him on a night out and we slept together. The next day it wasn't awkward at all.. I thought it would be but we spent ages talking and got on really well/had rather alot in common. So I kept speaking to him/him coming over and we continued sleeping together. He started asking me if I wanted to go to the shops together.. so we did that a few times and just got on really well. He told me he thought I was lovely, he agreed we get on really well.. but when I started to ask him what he wanted, I made it clear I wanted a relationship. And I've had this discussion with him two times.. the second time getting really upset about it when I was with him. He said he felt a bit pressured/hadn't known me long enough/doesn't particularly went a relationship.. then he saw I was upset and said look, next week lets go out for a meal to this restaruant we both like.

So hes invited me to do a couple things.. but I have quite a few reservations.. like the night we went for a meal he cancelled on me because of a friends birthday at last minute, then changed his mind and went ahead seeing me. And this hasn't been the only time hes made promises to see me and suggestions to see me and hasn't gone through with them. Its been about 5 weeks now and he still comes round.

I've never been round his place though.. seen him with his friends, I always just see him alone. When we were out together going to the restaruant he saw my flat mate and her boyfriend (his course mate) and got really stupid about it/tried to hide.. for some reason he gets weird about seeing his course mate when hes with me. But apart from that we have alot of similar interests, we make each other laugh etc. and I am not sure to do like he said and continue on.. and just see how things turn out, or just not speak to him any more and forget it? if I don't text him for a day or two he will text me, and he will ring me straight away if I ask him to ring. Hes always hugging me when I see him, or if he thinks I am upset hes quick to try and sort things out/make sure I'm ok. He hugs me goodbye/kisses me.. says 'I'll see you soon' but I just feel like this is going to end badly, he told me he wouldn't mess me around/the last thing he would want to do is upset me. But I am already getting upset because its kind of obvious I'm more into him?!