Replication of empirical results in Economics

Question

I was reading about the replicability crisis in Psychology and Medicine. Many psychology experiments, for instance, were being conducted on undergraduate students, with too low sample sizes, which claimed statistically significant results. And there was no attempt to redo those experiments with larger sample sizes by independent researchers. Many such results were generally accepted as true by the psychology community without proper scrutiny. It is only in the last decade or so that psychologists have seriously started to try to replicate many such studies with larger sample sizes, getting significant results in the same direction in only about one third of the studies, hence the crisis.

A similar replication attempt of experimental economics results published in the AER saw a two third replication rate, twice that of psychology.

But my point is, most empirical studies in economics are not experimental, but quasi-experimental. The data is often publicly available from government records etc. In such studies, their is no point of "redoing the experiment with larger sample size". What would replication mean in such studies: would it mean answering a similar question with different data and getting the same result? Even if the technique is not the same (for example if the first was a discontinuous regression on Swedish data to see the effects of class size on scores, can the second study be a Diff-in-diff on Brazilian data to answer the same question)?

And because usually the sample sizes in such quasi-experimental studies (even without replication) is usually quite large, can we say that in general replication is not an issue in Economics? Especially with the data being public. Or is my understanding of replication in quasi-experimental studies wrong?

Answer 1

In such studies, their is no point of "redoing the experiment with larger sample size".

This is simply not true.

1. More observations is always better. If you have some non-experimental or quasi-experimental research (lets say on relationship between debt an growth), and due to data limitations you are able to include only 30 western countries with 10 years worth of data, then it is actually extremely crucial that you check whether the relationships you found will hold once more data on more countries is avaiable.

2. Scientists can engage in some bad practices to get published such as p-hacking. Thus even if you do not have more data it is worth while to redo someone's else work to see if all steps make sense or if there is something off.

   For example, an author can claim that one control was excluded due to high collinearity with another control, auxiliary tests are unfortunately often not reported inside papers so only way to check is to actually replicate the study, and you might find that there was no collinearity and inclusion of that control made main result insignificant etc. There is so much of shady stuff people can do when it comes to methodology that being forced to share your code and having other people's literally retrace your step is the only way you can make sure there are no skeletons in the closet.

3. Replication broadly speaking is more than just simply retracing steps others done. Even in physics when you replicate experiment you might not do it 100% exactly in the same way if you think that the paper you replicate have some problem. For example, if you think results could be biased due to presence of air and the other experiment only managed to get 94% of air out of the test chamber, but you have better equipment and can get 99% of it out, you wont purposefully recreate the 94% conditions but you will try to get rid of as much air as possible.

   In similar fashion replication can be more broad than just retracing the steps. Did the authors used clustered errors but only had 10 clusters? Well then likely their clustered errors are misestimated, let's do replication with bootstrapped errors and see if inference changes. Did the authors use Diff-in-diff, but you discover they did not check for common trend assumption, you find common trend was violated? Well lets redo what they did but use synthetic control instead of DiD.

As you can see there are plenty of reasons for replication even in quasi-experimental/observational research.