depending on what type of retainers you're given and who your orthodontist is, you could end up being told to wear them all the time for a while or just at nights. the first time i had my braces taken off i was given retainers and told the only time i could take them off was for eating, consequently i didn't wear them and ended up with a second set of braces. this time i've been told that is has to be every night for the first 6 months then start reducing it - though i will have to wear them occasionally for the rest of my life
i can't remember it hurting much when they took the braces off, you just feel a lot of pressure, though i guess it could hurt for some people