There could be a lot of other reasons apart from accessory nerve palsy, perhaps go visit the doctor just to be on the safe side. Don't panic, even if it is that, the doctor will help you manage it
Another question: did you have any injury or trauma to the neck/shoulder?