What would you do?

I recently posted on here regarding rebuilding a friendship with a guy I slept with. We weren't in contact for 4 months, I essentially cut him off because I didn't think he appreciated the friendship and thought he might have been using me for sex.

He got in contact last month. Things were going okay, we talked through things and eventually we started flirting again. We both said we missed the friendship, he also added he missed the touching also. I told him I regretted the sex and didnt want to go there again whereas he said he was open to it (granted I was too).

One night a couple weeks ago, we went as far as having phone sex (yes, it was a dumb move), but it just came naturally and I was open to it at the time. It was a weak moment for me, it was exciting and I thought I actually missed the physical side of things. Over time I started to realise I didn't really miss the sex much , but I liked having him back in my life. I had alot of questions about what had happened which I asked him, I feel like I got a bit of closure which I needed. I started to take a step back and refrain from flirting with him.

He would hint about coming over to mine and I told him it wasn't possible, I told him flat out he couldn't on another occasion. He suggested we went out to see a film (but for personal reasons it isn't possible for me right now). My life is actually a movie right now, so I want to get through this tough time and not making any silly decisions in the meantime. I'm worried if we meet up and we are alone, we'll have sex again. The sexual tension between us the last time was alot and I just don't want to put myself in that situation at this time. So he asked me when we would meet up a couple days ago, so I asked what kinda setting he meant. He said he could come over and watch a film. He joked and asked if I wanted to just remain 'pen pals', I said I didn't mind it and laughed. He said he didn't want to stay pen pals.

I said "what if the same thing happens again? '' (the sex) it was me thinking out loud to be honest. He said" you're asking me this question? You already know I would enjoy it since I enjoyed last time. Maybe you should ask yourself that question or maybe you're overthinking 'again'. This was via text, but he seemed a bit annoyed. And that was that. He didn't reply after my reply, he hasn't sent me any memes or anything in days.

I feel sad about it and I don't know why. I have plenty of friends I don't see often, so I'm thinking he probably is hoping we'll just end up ******* again. Otherwise, why would he be annoyed and go silent on me?