Borders weren't invented by politicians. At the end of the day, humans are animals. Animals have terroritories.
That aside, I don't think free movement between civilised countries is a bad thing. Keeping out mirgrants from muslim countries makes sense though.