Owning a slave back then in America was basically the same thing as owning a house today: it was a system ingrained into the economy that was simply too big to fail, particularly in the Confederate states. So yes, as a white Anglo, whose family would in relative terms have moved up a class or three by emigrating, I would probably have owned a slave.
It is of course worth qualifying that not all slave owners treated their slaves with the sort of barbarism you see in lurid depictions of the period. While I would suspect correction was almost universal, not just for infractions and material damages but also merely for insubordination and work not done, I doubt that a majority of slave-owners were motivated by naked sadism.
Incidentally, I expect with almost no hyperbole that if you tried to take people's houses off them today post-Right to Buy you would have to have a civil war of some sort over it as well. At the very least it would involve the break-up of the UK and a level of violent insurrection completely unprecedented, at least outside of Ireland, in modern British history. Imagine. Paul and Sue from down the road huddled behind the Volvo stuffing rags into bottles of Cabernet Sauvignon.