It's a hard distinction, and I think it comes down to you. Evidently, it makes you uncomfortable and if he's a decent guy he should respect that. Maybe you could suggest going out together sometimes so you can be that girl for him. Whatever happens, don't start a huge argument or be confrontational over it, just draw a line in the sand and say in the future you would prefer him not to grind on with another girl.