Should you call it off if a guy talks about his ex this early on?

I've been in 2 long term relationships and they've both ended because of their ex; the first time he cheated several times and it was obvious really but I didn't want to face it for a while... the second he just admitted flat out that he still had feelings for her, and again I had suspected it.

Started dating a guy at uni, we were in bed together last week and he started talking about some girl he 'really liked' and she liked him too apparently (not just randomly, the conversation sort of led to it), how he'd been upset for the past few days because she had got together with someone last week.
I sort of brushed it aside.. but I'm seeing him again tomorrow and the more I think about it, the worse I feel... don't want to go through it all again. Is it worth just calling it off? Asking him flat out his feelings?