There are a few papers that I have seen that publish insiginficant results, such as responses (see Easterly's response published in the AER to Burnside and Dollars World Bank paper titled 'Aid, Policies and Growth). The aim of these papers, however, is to negate a positive claim which they do by producing insignificant results (in the hypothesis testing sense) using the same idea as the original authors, but using new data. However, I have yet to come across a major paper in its own right published in the top 5 journals that publishes results that are insignificant. This is a scary thought because when I do my own research, I always shudder before entering the regress y x command. Are there well known papers that publish insignificant results? Thanks!
1 Answer
This does not directly answer your question, but I will try to explain why I don't think that such well known papers exist.
If submitted I don't think that such a paper would be accepted by the top five journals. This is because the journals also compete to stay relevant, to give surprising information. The phenomenon is known as publication bias.
This was also a big problem for the medical community. The solution they arrived at is that they only accept trials if the trial and its hypothesis is reported to the authorities at the beginning of the trial. Because of this the result will be stored no matter what it is. Even though unsurprising results are unlikely to get much publicity, a researcher looking for them can find them. The same solution cannot work for economic regressions because running a regression on the same data always gives the same result. Hence false positives are not a result of lucky samples, but of tweeking the model to particular data.
-
2$\begingroup$ Could you be a bit more clear in your last paragraph? I feel I don't fully understand the contrast you make between the issue in medicine and the issue in economics, and how this modifies publication bias in the two fields. $\endgroup$ Nov 26, 2015 at 17:44
-
$\begingroup$ Medicine: Suppose you have a pill that has no healing effect. If you run 20 trials 'on average' one of them will produce results that are significant on the 5% level. Omitting the other trials gives a false probability estimate of effectiveness. If you redo the same trial, the pill is unlikely to have a significant effect again, because the next random sample is independent of this one. $\endgroup$– GiskardNov 26, 2015 at 17:57
-
$\begingroup$ Economics: You have some data and you are trying to build a model around it. If you run 20 regressions, each with completely different function forms, you will again 'on average' have one that provides data that is significant on the 5% level. The randomness here was in the function form, not the data. No one will argue that you should change your function form. Your model seems solid until new data comes around. You could of course have an obligatory 10-year waiting period before publishing. $\endgroup$– GiskardNov 26, 2015 at 17:59