What, exactly, is hurting you in this situation? He is friends with an ex. That's not unusual. It's actually fairly common for people to decide to break off a romantic relationship and stay friends, particularly when they have been together for a while. But in the absence of any evidence at all that he's cheating on you or has not been faithful to you (and you haven't highlighted any), there's no reason to be bothered by this. In fact, quite the opposite, because he's actually been entirely open with you about his contact with her and has communicated that to you in a way that he knows you'd be most receptive to it. That's a really good sign. Ultimately, he's actively decided to be with you. Not her. You. And has given you no reason to doubt that choice.
To me the whole 'casual' label is largely irrelevant. This is not a casual relationship, as much as you may want to label it as one. If it was, then you certainly have no right to tell him who he should be friends with. But even if I'm right and it's not, you still don't really have that right here. In any relationship you cannot be with your other half the whole time. They are always going to spend a good amount of time without you, and will, frankly, have plenty of opportunity to cheat on you if they want to. You cannot stop that. So you have to trust them. There are obviously things they can do to help with that, and there are times when you might reasonably not want them to put themselves in a particular situation, and indeed there are times when they should choose not to put themselves in a particular situation. But generally you have to trust that they will remain faithful to you. If you cannot do that and feel like you have to dictate who they spend time with, that isn't a great sign for the relationship. And I say that even when he's friends with someone that he has previously been in a relationship with, because it's common for people to still have contact with exes. Again, it comes to trust. For this to be sustainable long term, you have to trust him, and as I say, in this situation he's actually positively given you every reason to trust him as far as I can see.