If you're moving down as a family then the areas to avoid are the top end of Exwick, anywhere near Burnthouse Lane (it's down towards the bottom of Topsham Road), the top end of Mincinglake and also the main student areas which will be noisy. There's a few other not so nice areas but I can't remember them at the moment!
Topsham's nice, Pennsylvania (the bit north of Union Road) is also nice as is Pinhoe and Redhills. Generally they're a bit further out of town but the buses around Exeter are fantastic so it doesn't take much time to get into town (albeit the high number of old people round here make the tickets fairly pricey).
Compared to the rest of the country Exeter is incredibly safe. I did work experience at a place on the Salford Quays last year and went to Salford a couple of times and it didn't seem like the nicest place in the world! Mind you basing things on reputations is a bad idea...I've lived in Nottingham since I was born and never had a problem even though everyone think's its terrible for crime!