You have strong feelings for him.
He obviously doesn't have them for you. He just wants a shag.
If all you want is a bit of casual sex (emphasis on casual), I'd suggest go for it, but I bet you won't be able to leave it at that.
If you don't accept it will only ever be casual, it will end in tears for you. You will get with him, I'm pretty sure of it, and when you try to take it to the next level, he'll refuse, and you'll come back here and whine as if you weren't warned here and that you didn't see it coming.
"All guy's are the same..." I can hear you saying it now...
I suggest against it girl, but I doubt you'll listen. Setting yourself up for a fall.