Have a Google of Arousal Non-Concordance. It's very common, happens to every single person, and is where your brain is not in sync with your body when it comes to arousal. It's very common for boys/men to get erections when they are not turned on, but it is equally very possible for them to be turned on and not to have an erection. Indeed, it is perfectly normal for erections to come and go during a sex session. Unfortunately, largely because both men and women tend to think that the erection is the be all and end all of a man's sexual responsiveness and ability to give sexual pleasure, when a man loses an erection during sex it tends to trigger anxiety for both partners, as it has done here.
I think you know what the issue is here. I first had sex over twenty years ago and I can still remember how intensely stressful it is to put on a condom when you're not entirely sure what you're doing and/or you're struggling to maintain an erection, ironically because of how stressful the situation is. Equally, as the woman you're sat there wondering how to deal with this, and worried about whether he's attracted to you. Your endometriosis doesn't help because it obviously creates a further barrier to you enjoying sex, but this situation would have been difficult without that.
The answer is that you haven't ruined it, you haven't scared him off, and you are very much attractive enough. It's just that you're both inexperienced, and are anxious about letting the other one down. That's normal, but you need to take the pressure of each other. Sex isn't about impressing anyone. It's about enjoying the experience and enjoying each other. Focus on the pleasure of it and not how long he's maintaining an erection or how long either of you take to orgasm (if you do at all). Go in without expectation, talk to each other about what feels good and what doesn't, and have fun with it. You have so much to explore and so much time to do it in. Just go with it.