If you don't want to go, that should be your choice and they should accept that regardless of their own view on it. This is about you, after all. You could always suggest as a compromise to do something special as a family on the day though so they can celebrate it, if that's what they're going to miss? Fancy dinner or something?
Also, with regards to the people looking at you, your 'part' in the ceremony will actually be extremely short, so I wouldn't worry about that if you end up going.