It is hardly exploitation; we don't force them to work, they take the jobs because it is the best option available to them, it's better than working in agriculture. It is a stage of economic development that every country has to go through, for example, Britain in the coal industry, where the conditions 80 years ago were just as bad, if not worse.
China, for example, is growing at an extremely high rate and that will bring with it a higher standard of living. There will always be poor countries and we would not be able to enjoy the quality of life we enjoy today if it weren't for 'exploited' labour.
If we don't buy the clothes, that will put hundreds of thousands of people out of work- little money is better than no money. Also, handing out higher wages is not cost effective and would make investing in the east pointless.