No it's not rape, but he seems quite manipulative. You should not be with him. Even from seeing it in the best light possible, which is, that he has a higher sex drive than you, you should not be with him.
Ultimately, if it's all about sex drive incompatibility then you shouldn't give in, because you really don't want it. That's basically what going on: you eventually giving in, and because you have given in before he knows if he keeps on trying to convince you then you will give in to him once again. He probably even thinks in his own mind that he has 'turned you on' with his fingering. Don't let him believe this.
Rape has to be forced, and dropping your opposition and crucially consenting, verbally or otherwise, to sex is NOT rape. (Otherwise every lothario and player out there would be a rapist! Dicks yes, rapists no.) Just flip it the other way around, if it was a woman doing this to a man - trying to turn him on when he is not interested, until eventually he gives in and goes through the motions is exactly the same as your situation.
As a man, i've been in this situation myself: a bit sore down there and I'd had as much sex as I wanted or needed. But was with a girl who had a higher sex drive than me, and end up having to give in to pleasing her. Needless to say that relationship ended because it couldn't go on like that forever.
Basically, the guy is a bit of an immature idiot to see that you're not interested. Get rid. But do not report this as rape...please, for everyone's sake. He may be a dick, but no-one needs a false accusation of rape against them...which as you're not even sure if you've been raped, that is what it would be.