People are judged by their behaviour. If they had a history of flourishing and being a great economic advantage to their hosts then everyone would be queuing up to take them in.
We just use the information we have to hand to make the best judgments we can. Obviously religions tend to take issue with the use of information.