Economic Confirmation Biases

Question

I'm not sure if this is the right place to post this, since my question is a bit more general to all of social sciences (indeed sciences in general), but I figured since Economics is a social science and since the question I have is somewhat economics related someone here might have an idea. I am a statistician but work with a lot of economists who test various economic theories through observational data (not experimentation). Some of the "studies" I've seen carried out recently concern me because it seems, no matter the findings our outcomes, one particular researcher tries to tie the specific results back to a pet theory of his. In other words, the hypotheses are not clearly laid out in advance, but he looks at some data and if it shows 1 thing, he's say, Ah-ha, that is because my theory explains it. But if the same data showed just the opposite finding, he's still say, ah-ha, this is because my theory explains it! Specifically, let's say he compares 2 items for purchase that are identical in all physical aspects but one. If the price of item A is higher than price B, he's explain it with this theory. But if price of item A is lower than B, he'd simply twist his theory to to explain it as well.

Is there a name for this? It sounds like confirmation bias a bit, but not quite. I'm just curious if there is a name for this way of manipulating the findings to fit a narrative? Any ideas would be greatly appreciated.

Answer 1

Confirmation bias is somewhat different to the behavior you are describing. This expression describes the attitude of a researcher who interprets ambiguous data as confirming his initial hypothesis. For instance, researchers who believe that death penalty has a deterrent effect on crime would interpret any additional dataset as confirming this theory, whereas researchers who believe the opposite would interpret the same evidence as indicating that the effect is null.

The behavior that you describe might be called "post hoc analysis" or "post hoc theorizing". Scientific standards in experimental economics have been improving recently to mitigate this bias (among other mistakes/misbehavior). For instance, many researchers who do randomized controlled trials in development economics now pre-register their analysis plan (and even sometimes a theory with its predictions) before collecting the data: by doing so, they commit to a list of statistical tests that they will run, which prevents them from changing their theory/hypotheses (and even the question of interest) after looking at the data. Of course, as @denesp mentions, this cannot generally be done with observational data, except maybe in situations where the researchers need to go through an approval procedure to get access to the data, in which case a pre-registration of the analysis plan might be imposed as part of the application.

Answer 2

I think the general name for it is 'being a charlatan'.

Hypotheses need to be put forward before testing. This is unfortunately not common practice in economics, as we do not have trials like other disciplines. The commonly used historical data is frequently available in the secrecy of one's home, so no one can tell if I have already performed some tests on the data before putting my hypothesis forward.