Virginity is a word because of a bunch of old people who are all dead now that had their lives revolve around a father's confidence that some children would be biologically his.
Outside of that, it should be irrelevant. It's people bumping together. Really, not a unicorn. It also has nothing to do with love or happiness so if you feel that way it's an illusion unfortunately created by the world around us. It's just, not really a thing. Unless you make it so, which you seem to have done.
People not being attracted to you is also not a fair enough reason, maybe I am wrong here but if I wanted to find a guy to shag there were plenty, and I am a bit of a man, so can hardly imagine you genuinely not having someone. Then again why on earth would you want to bang a randomer...
What haunts you is the utterly unreal depiction of sex, much like the depiction of the human body and other such things that exist solely in the fantasy universe. Reality is way more ****, and virginity means nothing. Feeling insecure about the "status" of your body as dictated by others, and hence giving the media or society ownership of yourself, however, does mean something. Start there.