# Is there a reproducibility crisis in Economics?

I am an undergraduate economics student, and came across this paper by John Ionnadis on 'The Power of Bias in Economics Research'. the findings are quite bleak.

Some examples I have come across which worry me. From my studying of econometrics, this really doesn't surprise me. I don't think I've seen a paper, for instance, with test and training data, and an advanced undergraduate who did research for an academic over the summer told me that he basically messed around with parameters until he got the result he wanted. In fact, this undergraduate will be going to an extremely prestigious university for his PhD next year.

In economic history, which was the first area we read into the literature in first year, justifying the assumptions of OLS was always saying 'this should be plausible', for instance in a famous paper about settler mortality as an instrument for institutional quality. This sort of claim is hard to justify (e.g. settler mortality was probably influenced by prevalence of diseases like malaria, and who knows the broader implications of that).

Is there a strong case that Ionnadis is wrong?

• I hope Jan Höfler or someone more knowledgeable will give you a complete answer, but in the meantime take a look around the ReplicationWiki. – Giskard Feb 19 '20 at 20:55
• What would count as a "crisis" or "a strong case that Ionnadis is wrong"? (I think this is a perfectly legitimate question for the discipline of economics, but is not well-suited to this site.) – Kenny LJ Feb 20 '20 at 0:51
• @KennyLJ +1. Is there a SE site where this kind of question is appropriate? Cross Validated? – Art Feb 20 '20 at 2:16
• @Art: No or not that I know of. As stated on the SE tour, "This site is all about getting answers. It's not a discussion forum. There's no chit-chat. ... Not all questions work well in our format. Avoid questions that are primarily opinion-based, or that are likely to generate discussion rather than answers." – Kenny LJ Feb 20 '20 at 2:29

There is a general reproducibility crisis in most fields of science including economics. First economics is actually doing quite well compared to other social sciences for example this study shows 54% of studies that authors tried to replicate in psychology could not be replicated but this study for economics shows only about 40% failed to replicate. This being said 40% is quite a lot.

So I would say yes there is a reproducibility crisis in economics (although note what is and isn’t reproducibility crisis is to certain degree opinion based).

However, I think that examples which you gave are not the main reasons for this crisis.

The first example of that undergraduate students is basically example of conscious $$p$$-hacking. This certainly occurs but usually it is done by people who are not really good at research to begin with and are under extreme pressure to show some output, so they resort to this "cheating" in a similar way bad students resort to cheating during exam. I would not say that at good universities vast majority of researchers would consciously $$p$$-hack. However, note I on purpouse separate conscious $$p$$-hacking from sub-conscious $$p$$-hacking that can occur due to subconscious biases that any of us have which can also occur when you are simply too invested in some topic so you even without realizing become little less rigorous when creating your model. Unconscious $$p$$-hacking is much bigger problem because it sneaks upon you without even realizing that your doing something wrong.

In the second example, I would not even say that is an example of doing some incompetent statistical analysis necessary. In case of IV you always can in the end rely only on some logical story based on which you justify the exclusion criterion and validity of your instrument. Statistically, you can measure strength of the instrument using $$F$$-statistics from the first stage, and relevance of instrument from looking at the output and some auxiliary stats there, but there is no statistical test to date that would allow you to check whether instrument is really exogenous and only affects dependent variable through the endogenous independent variable. Furthermore, sometimes you cant find any perfect instrument or model specification etc. so you will just do your best and as long as you are explicit in the research findings that the results rely on some oversimplification or possibly incorrect assumption, the findings might be still useful. In that case even if the results would turn out to be non-reproducible I personally would not consider it a problem.

What is really at the heart of the reproducibility crisis is actually precisely what Ionnadis finds, that is most of the studies are underpowered. However, this is not due to some conscious attempt to make it so but because power of most statistical tests depends on the number of observations and in economics its often very expensive and hard to get more data.

Next problem is the publication bias. Even if you manage to get solid power ($$\geq80$$%) as this video from veritasium shows, using 5% significance and with having 80% power, and assuming that out of 1000 relationships only 10% are correct, even if you do everything completely by the book without any bias (from the perspective of scientist) or messing with the data, if there is a publication bias towards positive results, you will find that almost third of the published studies will not be reproducible at all (so now the 60% of reproducibility from that study about experiments in econ suddenly does not even look that bad). This problem is not easy to solve because if we would move to lets say 3/6/9 sigma like in Physics then just due to sheer lack of data almost no result in social sciences would ever become significant. Moreover all journals care about readership (since that usually correlated with getting more citations and thus higher ranking of the journal), so they have strong incentive to publish novel interesting research rather than replications.

So to sum up. Yes there is a reproducibility problem in economics (and most branches of science). However, its not necessary due to conscious p-hacking or incompetence. Sure that can play a role, especially at worse institutions, but generally the problem is low power, publication bias and subconscious biases. This is why the problem is so hard to solve. Low power is due to data limitations, its hard to do much about that if collecting data is expensive. Publication bias, can be addressed but every journal wants to publish `sexi' new findings, so its very hard to find proper incentives to do that (although things are getting better), and finally its very hard to deal with potential subconscious biases - you might get so much convinced about truth of some model/statement that you unconsciously p-hack by lets say trying too many different proxies for the same thing and focus just on the best results - this is probably the toughest problem to get around as it requires large amount of self-awareness and discipline.