37
$\begingroup$

I was reading on institutions, and I came across Acemoglu, Johnson and Robinson (AJR) paper on The Colonial Origins of comparative development: An Empirical Investigation, and this paper seemed so 'perfect' that spawned a lot of other papers based on the data of ancient settler mortality rate and how it was used by AJR as an IV for the current quality of Institutions in former colonies.

However, as I began searching more on this paper, I found out that the paper is in fact ridden with errors, specially on data treatment. (Albouy's 2012 comment on the same paper, published by The American Economic Review)

I would like to know, if, besides this paper on development economics, are there any other seminal papers, in any area of economics, that later on were proven to have crucial mistakes/errors that invalidated the paper conclusions?

$\endgroup$
  • 5
    $\begingroup$ How about we do as we do for books, one suggestion per answer? So people can vote on them individually. $\endgroup$ – FooBar Apr 25 '15 at 16:47
  • 1
    $\begingroup$ Note that even in the case of the paper you cite as being "ridden with errors", the original authors vehemently reject Albouy's comment. They open their reply to his comment with the extraordinary quote "Hither thou shalt come, but no further.". Likewise with the more famous and more recent Reinhart/Rogoff controversy. I am not aware of any empirical economics paper where the original authors ever got to the point of saying "OK we messed up." There are errors in every paper, but no author has ever admitted that their paper contains any fatal errors. $\endgroup$ – Kenny LJ May 7 '15 at 20:24
  • $\begingroup$ @KennyLJ That reply is to Albouy's first comment, it's not to Albouy's 2012 comment which was published by AER. $\endgroup$ – An old man in the sea. May 7 '15 at 21:01
  • $\begingroup$ @Anoldmaninthesea: Please take a look at the contents of the October 2012 AER issue and you'll see that the AJR reply I cited in my comment above was indeed a direct reply "to Albouy's 2012 comment which was published by AER". $\endgroup$ – Kenny LJ May 7 '15 at 22:21
29
$\begingroup$

My favorite example is the initial formulation of Arrow's impossibility theorem in the first edition of Arrows' "Social Choice and Individual Values" (1951). In the first edition, Arrow claimed that, together with 4 other conditions, the following domain condition

``The domain $\mathcal{D}$ is sufficiently extensive so that there exists at least one free triple of alternatives. (A triple is called free if all conceivable combinations of individual orderings of this triple actually occur in $\mathcal{D}$" (rewording from Blau(1957))

implied that there exists no social welfare function $S : \mathcal{D} \rightarrow \mathcal{R}$, where $\mathcal{R}$ is the set of all possible orderings (i.e. complete and transitive binary relations) over the set of alternatives $A$.

This was later showed to be false by Blau (1957) The Existence of Social Welfare Functions" Econometrica Vol. 25, No. 2 (Apr., 1957), pp. 302-313 who provided a counter-example.

Blau also showed (among other things) that the theorem could be corrected by replacing the above domain condition by the following condition

Universal domain : the domain $\mathcal{D}$ of the social welfare function contains every possible profile of preferences over the set of alternatives $A$ (with $|A| \geq 3$).

Arrow later corrected this mistake in the second edition of Social Choice and Individual Values (1963), and the formulation of Arrow's theorem using the Universal domain condition has now become standard.

This being said, the initial error in the first edition of Arrow's book was rather minor, and the solution proposed by Blau does not reduce in any strong sense the importance of Arrow's result and approach. Intuitively, the conclusion remains that on a vast domain of relevant economic problems, no social welfare function satisfies a set of rather basic and reasonable conditions.

So this might not be exactly the kind of errors you were looking for (definitely a seminal paper though!), but I like the example so much I could not resist posting it . If such brilliant people like Arrow make these kind of mistakes, I guess it takes a little bit of the pressure off for everyone else?

$\endgroup$
22
$\begingroup$

Kydland and Prescott's seminal paper on RBC theory uses a log-log specification on consumption-leisure preferences, arguing it is the only one that matches long-run constant share of working hours (one of the Kaldor facts).

This is false. In fact, there's a whole class of additively-separable utility functions (King-Rebelo-Plosser, which were discovered (published) in the same decade) where income and substitution effects from labor income cancel and do not affect the working hours decision.

Why it is relevant? Well, because their log-log specification gives them a huge Frisch elasticity, which is the only reason why they match labor. The capital stock (as opposed to investment) doesn't vary that much over the business cycle. It is then circular logic to get TFP shocks as Solow residuals from the data, feed them into a model where capital does not move that much, and observe that you get back $AF(K,L)$ (where $K$ is almost constant and $A$ comes from the data).

Don't get me wrong, it is still a very important paper, as it is the basis of most modern macro. But people at that time were astonished how good RBC fits the data. Well, if you don't fit labor data, the rest isn't that surprising.

$\endgroup$
19
$\begingroup$

Very influential paper by Angrst and Krueger (1991) used quarter of birth as an instrument for effect of schooling on earnings. Since compulsary schooling stops when you reach a certain age (so many dropout when they can). However it turned out that quarter of birth is not a good instrument, correlated with family background and thus also earnings.

http://web.stanford.edu/~pista/angrist.pdf

Edit as requested by 'An old man in the sea': take a look at a (Buckles and Hungerman, 2013), that show clearly that the date of birth is correlated with characteristic of the mother. So the instrument is not satisfying the exogeneity assumption (Edit2: since mother characteristics are clearly correlated with wage). enter image description here

enter image description here

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3777829/

$\endgroup$
  • 1
    $\begingroup$ Happy to hear that. For the record I think Angrist and Krueger's work is amazing, but today we know better than to use quarter of birth (in the us at least and for this purpose) as an instrument. Science advances. $\endgroup$ – snoram Apr 26 '15 at 19:16
15
$\begingroup$

In a 1929 paper, Harold Hotelling introduced what became the standard model of spatial competition. Two firms position themselves on an interval, which induces a certain demand structure, and then compete in prices. The model was influential and widely taught. The message was that firms diferentiate minimally, both locate in the center.

But in 1979 (!) paper, d'Aspremont, Gabszewicz, and Thisse pointed out that the model has actually no equilibrium under Hotelling's original specifications. There is an important discontinuity that Hotelling and everyone afterwards ignored.

$\endgroup$
14
$\begingroup$

This is freebee, but I'm gonna grab it: Reinhard and Rogoff (2010, AER pp) argued that there is critical level of government debt-to-gdp ratio at around 90%, arguing that countries that cross this level of leverage typically grow less.

Ignoring the whole point of correlation versus causality, UMass student + coauthors [reference needed] showed that this result only holds when

  • Using a specific, controversial weighting method that, iirc, was not emphasized in their paper
  • Specific and controversial sample selection (as above)
  • An error in their excel calculations that ignored some of the observations

Only when all these three coexist, one gets the 90% as a specific intolerance level. Otherwise, while higher debt-to-gdp ratios might correlate with smaller growth, there has been no study that shows a structural break / discontinuity at 90%.

$\endgroup$
  • $\begingroup$ I think something similar happened to the Alesina and Ardagna paper which argued that austerity was actually expansionary. I remember reading about it on Paul Krugman's blog, but I don't remember the details. $\endgroup$ – Keshav Srinivasan Apr 25 '15 at 17:05
  • 5
    $\begingroup$ I wouldn't call the Reinhard and Rogoff paper a "seminal" paper ... $\endgroup$ – user4239 May 28 '15 at 18:35
  • $\begingroup$ Our ReplicationWiki lists a number of replications of this study. $\endgroup$ – Jan Höffler Feb 12 '16 at 15:12
  • $\begingroup$ I just fixed the comment I had made a couple of days ago and that I now deleted to follow the rule that one should reveal the affiliation. One of the replications is what revealed the error, and I find important to show that the issues are not always so clear cut, there are various views and in the wiki you find the sources, also for the replies, so that you can build your own opinion. You wrote yourself "reference needed"... $\endgroup$ – Jan Höffler Feb 12 '16 at 15:36
9
$\begingroup$

Donohue & Levitt, 2001 showed a causal link between the abortion rate and the crime rate 20 years later. When abortion was legalized in the 1970s, a whole generation of unwanted births were averted, leading to a drop in crime nearly two decades later when this phantom generation would have come of age.

Legalised abortion accounts for as much as 50% of the drop in crime.

The problem with the paper was that the authors didn't do what they said they did. As part of their analysis, they subjected the data to a battery of tests. They claim to have controlled for a variety of effects which in fact, due to a coding error, they didnt. The Economist explains it as does The Wall Street Journal.

Donohue and Levitt did not run the test they thought they had—an “inadvertent but serious computer programming error”, according to Messrs Foote and Goetz... Fixing that error reduces the effect of abortion on arrests by about half, using the original data, and two-thirds using updated numbers.

Another error identified in the paper was that they used arrest rate totals instead of a population controlled variable.

These errors drastically cheapen the conclusion of a very prominent and controversial paper.

$\endgroup$
5
$\begingroup$

There is a book, The Halo Effect: . . . and the Eight Other Business Delusions That Deceive Managers (Rosenzweig (2007)), that highlights the methodological problems with a number of prominent management theory books including In search of excellence : lessons from America's best-run companies, Good to Great and Built to Last. They mostly amount to problems with selection bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.