A central tenet of libertarianism is the notion of free association–being able to socialise and enter into contracts with people of your choosing, with the corollary of being able to refuse to do those things too. The obvious criticism of this would be that one could then refuse to sell to people based on their skin colour, sex, orientation, etc., which has a historical precedent pretty much all over the globe. Does this make economic sense, though, or does it represent a market failure?
To my non-economist eyes it looks pretty cut and dry. Say Alice has a widget store and decides to only sell to people in group A. Bob also has a widget store and sells to both groups A & B. Surely Bob should have a significant economic advantage here, allowing him to lower prices and stay competitive with Alice? Even if some group A customers would refuse to shop at Bob's, presumably more would be attracted to Bob's store for its lower prices?1
Even if social or legal pressures meant everyone had to segregate, would this not present a gap in the market to be filled by group B store-owners, who would then sell to the underserved group Bs, with prices presumably more in line with group B incomes (if they are also economically disadvantaged compared to group A), and with the profits more likely to be redistributed amongst other group B businesses?
It's a cliche that the libertarian answer to everything is that 'the market will fix it', but this does seem like a sitation where that should happen, unless government pressure countered market forces (e.g. if it was made illegal for group Bs to even open their own stores or buy stock from wholesalers willing to sell to them). Would the libertarian counter-claim be, then, that the existence of economic segregation can only occur when backed by state force?