Yes it does happen. Orthodontists don't tell you to wear retainers at night for nothing, they tell you to do it for a reason.
Being in a similar situation, having to wear a retainer, I find that if I don't wear my retainer for one night my retainer wont fit me properly the next. That's how fast your teeth can move. In a space of 24 hours, I can see and feel a change in my teeth, god knows what it would be like after months of not wearing one.
Go and see your orthodontist and see what they say. As soon as your retainer didn't feel right you should have phoned them straight away. If you don't wear your retainer what was the point in you wearing braces? Your teeth just move back into their old position and everything you went through was a complete waste of time.