Over the years my commitment to banning those specific businesses has waxed and waned, but I still avoid businesses that I know to be terrible. I don't shop at Anthropologie and Urban Outfitters (the owner supports people like Rick Santorum) or American Apparel (skeeze and questionable employment practices), for example.
Still, it's dishearteningly impossible to really know your clothes are coming from a place that is both environmentally friendly and good to employees, even if the company is actually doing its best. We think a lot about the factories where the clothing is made, but what about what the clothing is made out of?
Man Repeller put together a pretty great breakdown of the environmental impacts of different fabrics. They don't address the factories at all, but this is still a valuable first step!
The basic rule from this post: Favour natural fibres as they are more renewable/sustainable, but avoid cotton (super environmentally expensive) and cashmere (the goats that supply it are eating too much and leading to desertification in Mongolia).
Of course, my favourite way to be more sustainable is to get as much second hand as possible. Thrift stores, consignment stores, and clothing swaps are the best! Why buy something new when you can get it used?