I don't know whether you'll count this as an economic argument, but it's about the role of competition in the allocation of scarce resources, so I think you should:
Suppose that some disease emerges that disproportionately affects the members of some ethnic group, or is concentrated in some geographic area, or is concentrated in some population that is either in or out of political favor with the current administration.
Suppose also that there's some pretty expensive procedure that helps to alleviate symptoms of this disease.
Under a single-payer system, one decision will be made about whether to cover the costs of that procedure. Quite plausibly, that decision will be either made or influenced by somebody with less than pure motives. Maybe an out-and-out bigot. Maybe a politician looking out for the people who support him. Et cetera.
I would much prefer that that decision be made independently by multiple competing insurance companies, so that people can choose up front to affiliate with insurance companies that they believe will look out for their interests.
If you think this is all just speculation, look at what happened when the government took temporary partial control of General Motors. Politicians immediately started thinking up ways to get General Motors to make decisions that would benefit their own constitutents.
General Motors is one firm, and the takeover was partial and temporary. Now imagine a takeover of the entire insurance industry, both total and permanent.
You might or might not believe the attendant risks are worth taking --- but you asked what the downside is, and that's a big part of the downside.