As much as I'm not that bothered by others supporting a non-local team, I would probably end up going to games of my new local team if I ever moved to a new part of the country.
As for it being hard to change teams, I don't know why you have to. Why can't you support two teams? I wouldn't stop supporting West Brom if I did move and start going to someone else's games.