2
$\begingroup$

One of the concerns that one can have regarding the use of RCTs for evaluating policies is that while when done in small scales, the RCTs may show the result in one direction, scaling the policies may involve general equilibrium effects that may possibly make the relationship of interest nil or even the opposite of what was found in the small scale RCTs.

Several possible examples, intuitively, can be thought of immediately.

But I would like to have some real world examples where this has happened. Links to any relevant papers that demonstrate this would be helpful.

$\endgroup$
6
  • $\begingroup$ What examples do you have in mind as you have said that "Several possible examples, intuitively, can be thought of immediately"? $\endgroup$
    – Amit
    Commented Apr 29, 2023 at 11:49
  • $\begingroup$ Umm one example can be: we do an RCT where the treatment is a job training programme. The treatment group's income increases as a result of the treatment when it is small, but as we scale it economy-wide, trainees lose the competitive advantage they have. $\endgroup$ Commented Apr 29, 2023 at 19:23
  • $\begingroup$ Another can be, when a new farming technique when tried on a small treatment group leads to increase in yields, but when implemented in the entire region, the land quality may degrade leading to probably even a fall in the average yield $\endgroup$ Commented Apr 29, 2023 at 19:27
  • $\begingroup$ Or a sales idea that when implemented by a small treatment group leads to rise in sales but when everybody adopts it, we are back to square one $\endgroup$ Commented Apr 29, 2023 at 19:30
  • $\begingroup$ You're right - such a thing can happen. The treatment effects of a policy on a small sample can be misleading if we aim to use the results to implement the policy on a large scale. To avoid this, we need to take into account the equilibrium while measuring these effects. $\endgroup$
    – Amit
    Commented Apr 30, 2023 at 23:39

1 Answer 1

1
$\begingroup$

I realize this is an old question but i was googling searching for the paper in question and found this and thought you might like it. An actual example of a paper where RCT and equilibrium disagree substantially about the efficacy of a program.

Denmark did a RCT i 2005 in two regions with more intensity in the active labor market policy. RCT study found a big impact. But this other study made later points out that the RCT violates SUTVA as the treatment and control group are competing for the same job openings. You could observe how people in the control group had their unemployment spell increased by the existence of the experiment by comparing them to neighboring regions with no experiment. The effect decreased substantially in size when these spill-over effects were taken into account.

https://www.iza.org/publications/dp/6748/estimating-equilibrium-effects-of-job-search-assistance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.