Should I be worried my boyfriend fancies/is sleeping with my mum?

I feel too embarrassed to say this to anyone in real life so here I am. My boyfriend and I have been dating for just over four months and although we hang out a lot he always seems to want to be at my house. Every time he's here he always wants to go downstairs where my mum's study is that she spends most of her time at. Last week I came back from the shop and walked into the kitchen, I could have sworn they were kissing and frantically moved away from each other once my presence was known. I brushed it off but he compliments her more than me at this point and more often than necessary I see them caressing each other. My mum has always been conventionally attractive, and I know this because in the past male friends have made comments about her. If my mum wasn't so passionate about us dating and him being around I would think nothing of it, but it really feels like something is going on. I know that the obvious option would be to just break up with my boyfriend, but it feels more personal considering it is my own mum. What do I do???