Would this upset you? (situationship)

How would you have felt if he had had sex with you last week and then told you he had sex a couple of days later with a woman you didn't meet? Would you have been upset by that?

I am female. I have 1 housemate, male and we have 2 dogs and share a car. We have been friends for a couple of years, living together for 8 months and hooking up for about 4 months. When we're getting together, he sleeps in my bedroom (I have a double bed and he doesn't), but otherwise we have our own bedrooms where we sleep and study. We have talked about a relationship but it's not the right time for either of us.
I don't celebrate Christmas and his family live abroad, so we are both staying in our house over the holidays. Yesterday we spent most of the day together and he was really affectionate, kissing my neck and hugging me a lot. He also slapped my butt a couple of times during the day. We have sex a couple of times per week and we had already discussed that he would make sure to buy condoms before the shops closed for Christmas. I had assumed we would have sex last night.
We took our dogs for a walk in the afternoon, and when we got back there was a woman I don't know sitting on a bench in the park opposite our house. He's known her for ages and they speak the same native language, so he went over to say hi. I followed. They started talking in their langauge and he did not introduce us. I went inside the house because I was cold, and they followed about 30 minutes later. They stayed in the kitchen speaking their language, so I went to my room and watched a movie.
When I went out to the kitchen for a snack, her bag was still on the couch but she wasn't there. They had gone to his room. He came out of his room a little while later, while I was in the kitchen. I asked what was going on. He said it was entirely chance that she was outside our house. She didn't even know he lived there (possibly true, we only moved here 3 months ago). He told me she was drunk and could not go back to her house because she rents a room in a family's home and they don't drink. She wants to sleep over at our house and she is already asleep in his bed. Our sofa folds out into a bed btw, and he only has a single bed which is uncomfortable to share. I asked if he wanted to sleep in my bed so that she could sleep in his and he said no, he was going to share with her because they have been friends for a long time, just like he and I have been friends for a long time. I was upset but trying to hide it. I can't remember exactly what either of us said, but he "accused" me of wanting a relationship. I was trying to explain why I was upset by his actions and as I was saying something like "we had sex 2 days ago," she came out of his room. She heard this and was upset and left. He is now angry with me and not speaking to me.
I'm now totally confused. I'm upset that he spent the day flirting with me and then planned to sleep with another woman (in my house!). I think that's a horrible way to treat someone. I don't want a relationship with him ever and I don't want a relationship with anyone right now. But we are friends and I feel like he used me. He thinks I'm being unreasonable and expecting too much. Am I being crazy for being upset about this?