Deaton and Cartwright make several criticisms of randomized controlled trials (RCTs).
My question is about this one in particular from a 2016 VoxEU piece:
Well-conducted RCTs could provide unbiased estimates of the average treatment effect (ATE) in the study population, provided no relevant differences between treatment and control are introduced post randomisation, which blinding of subjects, investigators, data collectors, and analysts serves to diminish. Unbiasedness says that, if we were to repeat the trial many times, we would be right on average. Yet we are almost never in such a situation, and with only one trial (as is virtually always the case) unbiasedness does nothing to prevent our single estimate from being very far away from the truth. If, as if often believed, randomisation were to guarantee that the treatment and control groups are identical except for the treatment, then indeed, we would have a precise – indeed exact – estimate of the ATE. But randomisation does nothing of the kind, even at baseline; in any given RCT, nothing ensures that other causal factors are balanced across the groups at the point of randomisation.
Similarly, in their accompanying 2018 publication, they write:
It is common to treat the ATE from an RCT as if it were the truth, not just in the trial sample but more generally. ...
the RCT strategy is only successful if we are happy with estimates that are arbitrarily far from the truth, just so long as the errors cancel out over a series of imaginary experiments ...
More fundamentally, we strongly contest the often-expressed idea that the ATE calculated from an RCT is automatically reliable, that randomization automatically controls for unobservables, or worst of all, that the calculated ATE is true. If, by chance, it is close to the truth, the truth we are referring to is the truth in the trial sample only.
I am having trouble understanding or appreciating the significance of the above criticism. It seems to me that the criticism here is analogous to the criticism that can be made of any estimate from a single random sample:
Suppose we get a random sample of a population, measure their heights, and compute a sample average height. Then Deaton and Cartwright's criticism or objection is simply that this sample average height estimate may not be generally true of the population as a whole; that we should be happy with this single estimate only if "we are happy with estimates that are arbitrarily far from the truth, just so long as the errors cancel out over a series of imaginary [samples]".
Does the above analogy describe all there is to Deaton and Cartwright's criticism? Or is there something deeper and bigger that I'm missing?