# What is the evidence that econometrics has empirical value?

Economies are extremely complex systems with many variables, not to mention the fact that they emerge from the interactions of complex beings. I agree that economies have certain underlying principles, but I remain skeptical of the overall value of econometrics as a science.

A truly useful econometrics would be a valuable tool in predicting the future behavior of the economy, particularly in predicting shocks like the recent financial crisis and Great Recession. But it did not happen.

If econometrics can't predict the future, how do we know it even effectively describes the past? What is the evidence that it has empirical value?

# Clarification

I regret any confusion this may have caused, but let me clarify what I'm looking for here. I regard this as an epistemological question. A better way to phrase it might be, "Can econometrics conform to the scientific method?" or just "Is it science?"

This is, at least in theory, a specific, answerable question, even if it is, as Alecos Papadopoulos noted in his answer, a matter of degree. (A view I share.)

This does not mean that specific examples of econometrics' successes and failures are not relevant.

• I have vote to close because the question is extremely broad. At the same time, it is a very important topic. But in order to expect some fact-based, referenced-based answers, rather than opinions of the answerer or of others, that the answerer will relay, I would suggest to narrow the focus. Choose your target: Is it Crises?(Econometrics is no Oracle). Is it some particular market? (there is a lot of _micro_econometrics). Is it International Economics? Etc – Alecos Papadopoulos Nov 27 '14 at 4:12
• So, in order to get an answer to this question, I should instead ask 10 or 15 more 'focused' questions? That would not actually answer this question. Respectfully, I consider that approach to be mildly absurd. – Gregory Higley Nov 27 '14 at 4:36
• Your question as it is requires an answer having the length of a book (and then some), it's that simple. Otherwise, you will get opinions, and this is not an opinion site. And yes, if you want to reach an overall conclusion to such a general question, you will have to break it down to more like 1,000 questions, find answers to them all, evaluate comparatively the answers, and then, see if you can answer with a "final", "overall", "all things considered", yes or no. Complex systems require complex tools and complex tools are not amenable to non-complex assessments. – Alecos Papadopoulos Nov 27 '14 at 4:54
• This remains unanswerably broad – EnergyNumbers Nov 27 '14 at 14:20
• I think the question is broad and mostly epistemological, but can and should be answered, otherwise we'll have the same question periodically again and again as a source of controversy in other answers to other questions. It's a valid (theoretical) question. The goal is to answer it by direct references to relevant arguments, not so much to convention or authority, I suggest. – user218 Nov 28 '14 at 11:04

To (slightly) paraphrase the OP:

Economies (Human Bodies) are extremely complex systems with many variables, not to mention the fact that they emerge from the interactions of complex beings (factors). I agree that economies (human bodies) have certain underlying principles, but I remain skeptical of the overall value of econometrics (medicine) as a science.

A truly useful econometrics (medical science) would be a valuable tool in predicting the future behavior of the economy (human body), particularly in predicting shocks like the recent financial crisis and Great Recession (severe illnesses or near death). But it did not happen.

If econometrics (medical science) can't predict the future, how do we know it even effectively describes the past? What is the evidence that it has empirical value?

Comment: There is a Present that needs to be dealt with, just like with human bodies.

To (slightly) paraphrase Milton Friedman:

I don't care how I predict, as long as I predict adequately.

Comment: But what do we do when our predictions are not adequate ?

Econometrics is based on Mathematical Statistics on the one hand, and on the assumptions made by Economic Theory on the other, which in turn are based on (imperfect) empirical observation, and are then led to their logical conclusions. In other words, Econometrics uses rigorous mathematics, induction, deduction and all the words dear to an epistemologist. Its epistemological foundations are solid as a rock.

What hangs on the balance is the observed validity of its abstractions. So the question is not whether Econometrics is a science, in the sense of whether it follows the scientific method or not. It does, fully. The question is whether its results are useful.

But what are the criteria in order to determine "useful"? Is it just adequate predictions? That would be a matter of disagreement between humans.

And is it a matter of a "yes/no" answer? Or is it a matter of degree to which it is useful? In which case, we have to somehow measure this degree (after we have agreed on the criteria), which brings us back to where we have to collect, analyze, assess, and debate the evidence.

• Good answer and interesting analogy. I should point out that medicine has a scientific testability that econometrics can only admire from afar: clinical trials, the daily operation of hospitals, the effects of drugs. Treatments that don't work are usually abandoned quickly. Econometrics has access to nothing even remotely comparable. – Gregory Higley Nov 28 '14 at 13:35
• @GregoryHigley Certainly. My main intent was to point out that "unquestionable predictive success", although perhaps the ideal final destination, cannot be a rejection criterion in Social Sciences (since it is not even in medicine), but only, exactly, the "ideal final destination". Meanwhile, we have to manage with what tools we have, while trying hard to improve them. – Alecos Papadopoulos Nov 28 '14 at 13:42
• Fair enough. My project isn't to discredit econometrics, nor to ask for "unquestionable predictive success", though it should be obvious by now I think it has far (far, far) less utility than you do, but it's not zero. We must "manage with what tools we have," though it's as if we're trying to mine for diamonds with a feather duster. At some point we may want to ask whether we are "managing" at all, or whether it's even a tool. – Gregory Higley Nov 28 '14 at 14:13
• +1 Although a lot of economics is more like physical cosmology, astrophysics or climate science, where the models are similarly complex and experimentation is impossible. The predictions of these fields are frequently inaccurate, but that in and of itself doesn't invalidate the "scienciness" of their methods. Those fields also aren't always immediately "useful", but make modest improvements to generally inaccurate models over time. – jayk Dec 1 '14 at 19:41

The first thing to say about this question is that it is important to look in the right places for good econometric predictions.

Physicists cannot reliably predict a wide-variety of quantum phenomena. Does this represent a failure of physics? No. Quite the contrary, we know (thanks to the work of a series of high profile physicists) that such phenomena are fundamentally impossible to predict.

In economics, too, there is good reason to think that certain kinds of shock or major economic event are virtually impossible to predict. Suppose that it was possible to predict the timing of the next financial crisis/recession, and that the next such event is predicted to happen tomorrow. Armed with such a prediction, firms would begin to reduce their capacity in readiness for the coming contraction in demand, traders would start to sell shares in anticipation of the coming stock market crash, households would reduce their spending in readiness for the expected contraction in their income, etc. In other words, anticipating the predicted recession tomorrow causes rational actors to take all of the actions necessary to induce a recession today: earlier than forecast! Any ability to accurately forecast the timing of such events is, therefore, to a large extent self-defeating.

Once one looks at variables that can realistically be predicted, virtually every field of economics is littered with examples of well-performing econometric models making predictions that are, by social science standards, excellent .

• The question, which the OP chose to leave unaltered despite the suggestions that it was too broad (and enough members of this site agreed with him to re-open the question), requests evidence. This answer says only that "virtually every field of economics is littered with examples of well-performing econometric models..." etc (the last sentence). Is this considered evidence? – Alecos Papadopoulos Nov 27 '14 at 17:48
• @Ubiquitous The comment was really addressed to the OP, in the hope that he will understand the impossibility of its question. Essentially such a question is an ideal invitation for the Chat, not for the Q&A space. – Alecos Papadopoulos Nov 27 '14 at 19:01
• @GregoryHigley Compare the confidence intervals we get away with, compared to hard sciences which can use real lab experiments. Its many orders of magnitudes... just look at the number of $0$s before the likelihood of the Higgs-Boson not existing. – FooBar Nov 27 '14 at 19:20
• One way of looking at it is that maybe econometrics has overstepped its role. The models and regressions that we have are too simplistic and yet too bold (they attempt to encompass a variety of phenomenon) to the point where our extrapolations are meaningless. – rosenjcb Nov 27 '14 at 20:15
• @AlecosPapadopoulos I'm sorry, Alecos, but I do not necessarily request evidence. You simply haven't grasped that I regard this as an epistemological question. It might be better phrased as, "Can econometrics conform to the scientific method?" A proper approach is not to perform a meta-analysis of its alleged successes or failures on a case-by-case basis, but rather to analyze the methodological assumptions that underlie econometrics to determine whether they conform to what one would expect of science. – Gregory Higley Nov 28 '14 at 9:15

Well, stackexchange administrators believe that comments should not be used for anything remotely like an answer, I am moving my long pair of comments from the day before to an answer (and since the OP upvoted them).

The question really seems to be to be asking for epistemological arguments, yes?

Econometrics, I would argue, has been most successful in economic demography and economics of education, where adequately controlled data leads (all classes attended, job placement, sufficiently large samples to satisfy statistical assumptions, with detailed relevant background information, etc.) have led to meaningful conclusions. That is the place generally to look for successful econometric explanations.

In general, however, it's rather difficult to be impressed by claims along the lines that econometrics replaces coherent theory or is somehow more primary in explanation of facts. Consider the well know arguments (due to the W. Clifford and E. Mach) against a theory of history and history as a source of theory: --- no well-constructed experiments => no valid "law" inferences. (Typically, we simply don't have accurate preferences data to control historical data with, etc., etc.)

Also, without theory, demonstrated by other means, one cannot decide which measurements or facts are relevant for collection and testing of another theory. In the end, the reason a theory is confirmed by facts or discarded, depends partially on another theory. The experience would not speak for itself, although in some physics, where we can control for almost everything, it may. (Although physics is generally highly theoretical too.)

A shear stress test is the same shear stress test for everybody. But whether a farmer who contributed food to feed Wellington's army contributed to the outcome of the battle of Waterloo (this a factual question) would covary in answer with whether theoretically the food he contributed (if he did and what is a factual question too) is relevant to the economic dynamics. Otherwise, we get spurious and purely coincidental correlations that appear significant.

Also, there are no constants in economics --- because agents learn and preferences change. To say: A explains the dependent variable quantity B with extent C, if true, may be true by historical coincidence, without a strict reason. For, to say that it always has relation C to B, involves the assumption that people don't learn (which is false).

So I would argue epistemologically that more mathematics is required in economics, not less, which per se means less econometrics, if we aim to explain experience.

EDIT

The underlying evidence for the premises of theoretical arguments are (a) physiological-psychological results of well constructed experiment. (Extinction of the orienting reflex, say, corresponds to Gossen's Law of Satiety.) Or (b) tautological thought experiment. Not historical experience, the basis of econometrics, which is a higher-order analysis.

"A if and only if B" claims cannot be falsified by historical experience; that may only falsify "A does not exist" claims. Science, as C. Wolff is famous for defining, is the set of proven or provable claims (at least to the extent, as K. Popper argued, that all known alternatives are falsified). The purpose of science is explanation, for systematic and justified prediction (not merely coincidental prediction).

A classical analysis paper on what is required for knowledge (the ability to systematically predict, not merely guess): Gettier E, 1963, Is Justified True Belief Knowledge, Analysis 23(6),121-123. An open link. Medical guesses, as opposed to systematic physiological study, would be pragmatic belief, as opposed to science, in the language of Kant (Critique of Pure Reason, 2nd edition).

@Alecos has already mentioned one of the two points I wanted to make, here is my extension.

Econometrics has no value per se, it always comes from the data we use it on. In microeconomics, with many studies that are natural experiments, quasi-experiments or where we are somewhat sure that we are not capturing endogeneity, we could estimate many interesting parameters. One of the standard references would be the many wage regressions that have delivered fairly consistent and reproducible estimates.

On the other hand, in Macroeconomics, we have no hope of getting clean identification. If you assume every country to be identical, all we have is a panel of around 60 panels and 30-50 periods of observations (mostly of aggregates), depending on what you want to observe. Moreover, these are often not perfectly measured, and most importantly we cannot create random variation here. Of course, when data fails, one can try to tweak the estimators, but even with partial identification as some VARs do it, we can only hope to get short-run identification, and still have to worry about endogeneity.

Yes, I agree. Economists have not consistently and univocally warned us about the financial crises and other macroeconomic shocks. But this does not necessarily mean that the structure that macroeconomic models assume are bad, nor does falsify any other section in Economics.

This is good news for Microeconomics: We can somewhat rely on Econometrics to reject and accept theory. It is bad news for Macro: We can almost never reject plausible models just looking at the data.

• Of all the responses, this one is the closest to my view, except I think I am still more pessimistic than you. "This does not necessarily mean that the structure that macroeconomic models assume are bad." Doesn't mean they are good either. How would we know? – Gregory Higley Nov 29 '14 at 15:41
• It seems to me that we reject macro models all the time. Consider various consumption based asset pricing models (these are definitely "macro" in scope). They produce falsifiable predictions and the data has shown that the models are definitely wrong. Just because identification is near impossible in macro doesn't mean macro doesn't produce falsifiable hypotheses. – jmbejara Nov 30 '14 at 13:26
• Sure, I should probably be a bit more careful with how I phrase that. I might look into it later again. But again, also with these models, we can argue that they are right at the core, but they lack an extension to become more correct, whatever that means. For example, the preferences that include past consumption . Im not saying that that is necessarily what I think is correct, but for many falsifiable prediction your model has, you can make up a story/extension that explains it. – FooBar Nov 30 '14 at 15:01
• There are some predictions that are so much off-the-hook that you just outright reject the model, but its not as easy as it is in lab-based science, even with falsifiable micro predictions. – FooBar Nov 30 '14 at 15:01
• > But this does not necessarily mean that the structure that macroeconomic models assume are bad, That's true, but it doesn't take a lot of work to show that most macroeconomic models don't hold up to real world data, and it's equally easy to point to their asinine assumptions as the culprit. Macroeconomists love to handwave away 100+ years of solid psychological and sociological research. – Nate Vomocil Dec 1 '14 at 19:45

Econometrics is nothing more than a fancy name for statistics. Namely, how do we apply statistics and economic theory together to get a clearer picture of reality?

Is statistics scientific? It can be when used properly. Statistics is merely a tool of science, in exactly the same way that econometrics is a tool of general economics. It is a tool that can be used for good science and for shit science.

• No one doubts the mathematical utility of statistics. It is the inputs that are the problem: Whenever the inputs are large aggregates of conscious beings with (potentially) very different subjective preferences, possibly across times, places, cultures, etc., when the number of free variables is very large, it is an open question whether there is much to be learned, or whether the discipline of statistics is even applicable. – Gregory Higley Dec 1 '14 at 20:18
• Indeed, but the answer to ""Can econometrics conform to the scientific method?" is "Yes, it can." – Nate Vomocil Dec 1 '14 at 20:34
• If econometrics = statistics, the answer is clearly yes. But that is pretty uninteresting, almost tautological. If econometrics = statistics + "economic data as inputs", the answer is "no" at the macroeconomic level, and more muddled at the microeconomic level. – Gregory Higley Dec 1 '14 at 22:10
• @GregoryHigley have you ever done any econometrics? Research level, not off-the-shelf modelling? I am asking because I am curious whether your hypothesis about uselessness of econometrics (micro and especially macro) is based on personal experience, or just the impressions? – mpiktas May 26 '16 at 11:26