Firstly, I think you should try and put your insecurities aside because however **** you're feeling about it, he is feeling a hell of a lot worse.
Feeling pressured into performing can cause guys to ejaculate too quickly, or even lose thier erection. I think claiming he has "erectile dysfunction" is pretty extreme! Are you the first person he has had sex with, without a condom? If so, it can take more than "a few weeks" for him to adjust to the heightened sensitivity.
Going soft inside of you; think about it. The last few times you've had sex he has came to quickly. He is worried that he is going to do this again & therefore he isn't focussing on sex and enjoying it, his just plain worried.
Oh and no matter how much your past boyfriends managed to get it up quickly after ejaculation - not every man is the same. You should accept this and not compare him to your ex's.
Don't give up, try not to put pressure on sex by planning it (By this I mean, dont text him saying you can't wait to **** him later as this will get him worried the second he receive this text! Make sure sex is random)
If you don't want him to come to quickly, alternate between things. Have penetrative sex for a little while and when his getting too excited pull off of him and play for a little bit with foreplay.
Worst comes to worse... slap a condom back on and see what happens.