Understanding this particular criticism of RCTs by Deaton and Cartwright

Question

Deaton and Cartwright make several criticisms of randomized controlled trials (RCTs).

My question is about this one in particular from a 2016 VoxEU piece:

Well-conducted RCTs could provide unbiased estimates of the average treatment effect (ATE) in the study population, provided no relevant differences between treatment and control are introduced post randomisation, which blinding of subjects, investigators, data collectors, and analysts serves to diminish. Unbiasedness says that, if we were to repeat the trial many times, we would be right on average. Yet we are almost never in such a situation, and with only one trial (as is virtually always the case) unbiasedness does nothing to prevent our single estimate from being very far away from the truth. If, as if often believed, randomisation were to guarantee that the treatment and control groups are identical except for the treatment, then indeed, we would have a precise – indeed exact – estimate of the ATE. But randomisation does nothing of the kind, even at baseline; in any given RCT, nothing ensures that other causal factors are balanced across the groups at the point of randomisation.

Similarly, in their accompanying 2018 publication, they write:

It is common to treat the ATE from an RCT as if it were the truth, not just in the trial sample but more generally. ...

the RCT strategy is only successful if we are happy with estimates that are arbitrarily far from the truth, just so long as the errors cancel out over a series of imaginary experiments ...

More fundamentally, we strongly contest the often-expressed idea that the ATE calculated from an RCT is automatically reliable, that randomization automatically controls for unobservables, or worst of all, that the calculated ATE is true. If, by chance, it is close to the truth, the truth we are referring to is the truth in the trial sample only.

I am having trouble understanding or appreciating the significance of the above criticism. It seems to me that the criticism here is analogous to the criticism that can be made of any estimate from a single random sample:

Suppose we get a random sample of a population, measure their heights, and compute a sample average height. Then Deaton and Cartwright's criticism or objection is simply that this sample average height estimate may not be generally true of the population as a whole; that we should be happy with this single estimate only if "we are happy with estimates that are arbitrarily far from the truth, just so long as the errors cancel out over a series of imaginary [samples]".

Does the above analogy describe all there is to Deaton and Cartwright's criticism? Or is there something deeper and bigger that I'm missing?

Answer 1

Without having read the full piece I'd say you are mostly spot on, but for a subtle difference. The objection runs a bit deeper than your rephrasing of it, where you refer to single estimate. The thing is that there are way more than a single estimate this applies to, and therefore you are more likely to be off.

The point is that for RCT to establish causality both groups have to be equal on all other characteristics except for the treatment. By doing assignment randomly one can expect that in a large number of potential samples for all these aspects the independent samples' averages will be close to the population means for these characteristics.

The problem is, as you rightly point out, one may be unlucky and have samples where the sample averages are not actually close to the population means. The additional problem is that sample averages have to be close to the population means for all variables that matter, but 1) you don't always know which variables matter and 2) given the large number of variables that do matter you'll always have one or two where you are "unlucky" in terms of the sample averages not being close to the population mean.