Talking about who "the rightful owner" is is dumb, and doesn't really help us much anyway. If we determine, for instance, than the Native Americans are the rightful owners of the territory that now makes up the USA, does that mean we throw out everyone else living in it? Of course not. We can say that they were historically wronged, sure, but we can't undo that now.
The more important framework is one of rights. What we have at the moment is one regime which governs all the land between the river and the sea, under which there are several different grades of rights and citizenship.