We can't. By both the international legal refugee conventions that we are signed up to, and EU law on asylum seekers, refugees have a right to family unity. In fact, if the family is still together, then the father is almost certainly what's called the "principal applicant" for refugee/asylum seeker status, while his wife and children hold "derivative status" as long as they are his dependents.