Ones hymen can get broken through all sorts of normal activity including exercise and horse riding. There's no way to tell if you don't tell them. More to the point would you really want to be with someone that's going to think less of you because you acted as a human being when you were younger.
I suppose you need to weigh up what you want. If my options were between a loving partner now or an unknown fantasy element in the future I'd go for the former (this doesn't mean that you need to jump into bed with him).
I'd also talk to your family about how you are feeling, if they don't care or are not open to this... then frankly I'm not sure I'd worry too much about 'letting them down'. Make safe and sensible decisions and forgive yourself when you make a mistake.
You're only human and have nothing to be ashamed or embarrassed about.