The relation between econometrics and machine learning

Question

In my understanding the utility of econometrics versus machine learning is generally argued to be the research of causality.

If I'm not wrong, when we put aside experimental economics, the golden standard to prove causality is to use Intrumental Variables. But in practice it has been showed that finding a relevant IV is often not possible, and authors end up taking the best "IV" they can find.

Problem: This variable has a high chance to not in fact be exogenous, but we can not verify it when the IV is dependant of variables like happiness, for example. Since we have no way do verify those dependances, science stops here.

SO I am wrong to say that in most cases econometrics can not scientifically establish causality? In that case why using traditional econometrics instead of modern machine learning techniques?

Answer 1

I.) 2 Principles of econometrics can potentially be useful compared to Machine Learning. (see Hal R Varian 2014 Paper : https://pubs.aeaweb.org/doi/pdf/10.1257/jep.28.2.3)

A.) As you suggest the search of causality is one advantage but unlike what you think, even if causality sometimes could be tricky to measure it remains very useful and functional.

But first, when you suggest that Instrumental variable is the only available tool which work for most of the cases and allow causal inference, i think there is a few more techniques that could still be applicable to measure causality in response to a treatment, a manipulation, or an intervention and still be relevant (unlike natural experiment as you precise because of its restricted direct applicability to most of the cases) in most of the situations such as:

● explicit experiments ● regression discontinuity ● difference in differences ● structural estimation

For instance you can investigate causality with theses techniques or instrumental variables because even if those techniques are subject to bias and correlation issues (all the more since big data because the increasing size of datasets limits the usefulness of instrumental variable methods, depending on instrument strength and level of confounding), it still give a hint to investigate causality. In a way that, for instance, it enables the detection of conditional counterfactual (an if-clause which is contrary to fact) while also check the existence of potential selection bias.

What is a Conditional counterfactual:

If it is raining, then he is inside. : Rain = Indicative Variable

If it were raining, then he would be inside. : Rain = Conditional Counterfactual variable

Hence, the better predictive model you have for the counterfactual, the better you will be able to estimate the causal effect. Thus, even though a predictive model will not necessarily allow one to conclude anything about causality by itself, such models may help in estimating the causal impact of an intervention when it occurs. Because it can highlights a bunch of conditional counterfactual which could be use as potential causal Variables to run the test on, in order to eventually determined causality in a given dataset to do good decision making (clean up from confounding issues). If not, at least, it gives you some clues to conduct a deeper investigation to understand the problem : anything wrong with the theory ? anything wrong with my econometric model ? anything wrong with my data ?.

For more insight about the pro cons of differents causal inferences techniques in econometrics :

https://www.jstor.org/stable/pdf/44234997.pdf?refreqid=excelsior%3A18bacfd86299dcf19e7f1f13d9c52022

B.) Except causality, model uncertainty is another advantage of Econometrics compared to ML.

Probabilistic foundation of econometrics is a strenght in a way that it allows interpretability of most of the models and their parameters (avoiding black box phenomenon) and give a quantisation of uncertainty (with confident intervals ) . The goal is usually to show that the estimate of some interesting parameter is not very sensitive to the exact specification used : how an estimated parameter varies as different models are used.This question illustrate a simple form of model uncertainty. In this period of “big data,” it seems strange to focus on sampling uncertainty, which tends to be small with large datasets, while completely ignoring model uncertainty, which may be quite large. One way to address this is to be explicit about examining how parameter estimates vary with respect to choices of control variables and instruments.

II.) On the contrary Machine Learning techniques could also be useful for data analytics in social Science.

A.) Parameters Selection, Model validation methods in ML can improve traditional econometrics models

Researchers in machine learning have developed ways to deal with large datasets and economists interested in dealing with such data would be well advised to invest in learning these techniques. For instance, Web Mining methods could discover new usuable explanatory variable. Cross validation should identify a non-linear effect or a forgotten cross effect. Model Validation should detect when a model is mispecified and thus allows a better specification of an econometric model and on the whole reduce omitted variable bias and error. As an example, recent litterature in finance focus on Garch models (traditional times series models) improved with Neural Network to better predict volatility and asset prices.

For more insight about the usefulness of ML in econometrics check out this paper by Arthur Charpentier : https://arxiv.org/pdf/1708.06992.pdf

B.) Causal Modelling began to be a concern and a field of research in ML. Which means that in the long run ML could potentially overcome its own flaws ( like being exclusively focus on the fit ) and outperform Econometrics.

Some Theoretical computer scientists, such as Pearl (2009a, b) have made significant contributions to causal modelling in computer science (by extension machine learning):

(see: Causal Inference in Statistics: A Primer Wiley, 2016 Judea Pearl et Al)

Pearl defines counterfactuals directly in terms of a "structural equation model" – a set of equations, in which each variable is assigned a value that is an explicit function of other variables in the system. Given such a model, the sentence "Y would be y had X been x" (formally, X = x > Y = y ) is defined as the assertion: If we replace the equation currently determining X with a constant X = x, and solve the set of equations for variable Y, the solution obtained will be Y = y. This definition has been shown to be compatible with the axioms of possible world semantics and forms the basis for causal inference in the natural and social sciences, since each structural equation in those domains corresponds to a familiar causal mechanism that can be meaningfully reasoned about by investigators.

However, it appears that these theoretical advances have not as yet been incorporated into machine learning practice. Except in some recent research paper :

http://www.nasonline.org/programs/sackler-colloquia/documents/athey.pdf

As a Conclusion, in my opinion and as things now stand, Econometrics as more valuable input in social science than what machine learning could bring. So it is still adequate to use econometrics to conduct data analysis instead of ML techniques.

Answer 2

My view coincides with the introduction to your question. Namely, a) Econometrics is mostly concerned with causality b) Machine learning is mostly concerned with fit.

But for the remaining part, our views depart. Here is why:

a) IV and other quasi-experimental techniques) are not the only way to test for causality. The alternatives are i) experiments ii) structural estimations. In both cases you apply econometric machinery although you will mostly use a simple OLS in the first case and bayesian/GMM/Maximum Likelihood things in the second. Compared to experiments, it is probably less clear how structural estimations help and here I come to the second point;

b) As in any science, economists build math models to show how things work. The problem is that societies are very different so we have to develop many models for many social contexts. So naturally, a questions arises: how to define, which model is an appropriate in given circumstances? Here is where the econometricians could help, because econometrics is suitable for discriminating between working and non-working models. There are different ways to show it.

With (the now so popular) quasi-experiments, you show that the hopefully causal link either exists or does not and its magnitude is $\beta_i$. A simple story with a model, which is linear in parameters ($\beta_i$ does not depend on time, social strata of the subject etc).

Now, what if your data does not fit into the shoes of the approach and doing an experiment is impossible? You can go for structural estimation, where you assume that the model XYZ - e.g. the Cobb-Douglas production function - is a good description of the real world. With this assumed, you ask how would the parameters will look like. So you take your non-experimental data and forcefully put them into the model you have and estimate the parameters.

How does it help to establish the "truth"? One way is to look at how reasonable and stable the coefficients are across time and studies. For instance, suppose you estimated the parameters of the Cobb-Douglas production ($Y = AK^\alpha L^{1-\alpha}$ s.t. $0 <\alpha < 1$) function and your coefficients for the log-log regressions are $\beta = [13, 0.8, 1.5]$ (these numbers are absolutely fictional). In this case you have a solid reason to conclude that the model does not fit the context (maybe the industry) you are studying because you got $\beta_2 = (1 - \alpha) = 1.5$. Notice that in this case $\beta_2 > 1$, which is nonsense given that $\alpha$ is within range of $(0,1)$. What I find great with this approach is that it makes you think in both directions: anything wrong with the theory? anything wrong with my econometric model? anything wrong with my data input?

People in Macro and Industrial Organization frequently follow the approach (their tools differ though) because they have limited abilities to experiment in the field. Otherwise, when you do follow the prescriptions of the quasi-experimental literature, you can investigate only a little subset of problems, which are crucial for our understanding of how economies work.

This is in my opinion the main point of the Deaton's critique with respect to the quasi-experimental approach to causal inference. It turns researchers into people, who are looking for a problem that fits the tool and care about the context inasmuch randomization is credible. As a result, you can publish studies on labor economics, make conclusions about political economy, analyze sports data and evaluate development policies in poor countries all at the same time and do not care about the underlying mechanisms. This approach, however, does not help too much with the model selection. Maybe the true relationship is linear, maybe not. But when instruments are strong and you use the right words in the section on identification strategy, it does not matter for the publication. This is probably not that bad per se, but Deaton is worried that the approach tell little about which models work and what are the values of the fundamental parameters (Check the response of Imbens though. Both things are good readings).

Why are the parameters important? Let's take an example from physics. In physics you make experiments, measure inputs and outputs, and obtain the coefficients. If we want to make a prediction about how fast the stone falls in a new place, we would take the previously obtained parameter estimates, plugged the input data and forecast how fast a stone would fall from a certain distance in a new environment. If you know how the new environment differs from another one (defined by the model!) you can use the coefficients you got, plug them into a model and get a credible prediction.

In economics, the values you get from the quasi-experiments won't help you to do the same thing. Think of a development program the World Bank started in Eastern Europe and want to apply in South Africa. Assume you had a credible inference with a super-fancy RDD strategy. Fine, you got your super-significant $\beta_i$. It is clear though that the impact of development program in Eastern Europe won't be the same as in South Africa because the context (environment) differs. So using the $\beta_i$ directly won't work. But can't we just somehow adjust the values and make a reasonable prediction? Well, since we don't know what on Earth the $\beta_i$ really is and how two models really differ, we don't know what kind of transformation to the $\beta$ we need to apply. So we know something for Eastern Europe but cannot use the numbers for other places. That's a pity right? Cause you did a good job, but cannot generalize its results. Structural econometrics can be explicit on what the coefficients - in terms of the model - mean and how to use the values when you transfer them to another environment. The price of it are stricter assumptions on the relationship between the variables and the structure you - as a modeller - impose on the error term.

c) In my opinion, machine learning is a valuable tool to collect data. A good example is the current stream of papers on protests and political economy. With internet you get access to a lot of unstructured information like texts. The ability to extract, process, and produce insights from it it using machine learning techniques enables you, for instance, to evaluate the sentiments of the electorate and how it affects future political outcomes. So in a sense ML is a good tool to save your time to create datasets to study novel problems or tackle the omitted-variable bias.

Answer 3

So you are right. It is extremely difficult to prove causality in economics. Using an instrumental variable is a good way to do so. I think you might be a little confused about the difference between "machine learning" and Econometrics.

Machine learning works in 2 ways:

1) You have a massive data set with the correct answers already keyed in. You split the data set in 2 into a test and train set, then run programs to create a function to identify the already correct answers given a variety of variables within the train set. You then can use the currently unseen test set to see how accurate the function that the computer has to see how accurate it is. A good example of this is computers figuring out which hand drawn number represents what real number because you can feed in pre-existing data sets.

2) You have no data but a fitness function (a function which defines correctness). You create your own data by having the computer randomly set the function that turns data into the output and then put the many random functions into a situation. After the simulation finishes then you see which functions performed the best and modify all the functions to be more like that. Over thousands or millions of iterations the functions slowly "tune" the functions so that the give the correct answer given the repeated situations. A good example of this would be self driving cars. Some researchers have literally loaded computers up into grand theft auto to allow it to practice.

Both cases have very limited applications to economics. In the first scenario you would need to have data sets on thousands of economies over thousands of years all with differing data. We certainly do have plenty of economic data but when you think about it we only have 195 countries to look at and there is only 1 global economy. We also only have around 100 years of good economic data. More over the relationships between variables seem to change. Conventional wisdom used to be the low unemployment led to inflation which doesn't seem to be the case anymore in the U.S. economy.

With the second machine learning practice there is no way to simulate the entire world economy accurately enough to test how different policies would impact the world economy due to not strong enough computer power.

Econometrics uses instruments because it allows for a natural experiment. You obviously can't just run experiments in the economy to see what happens (ex: What would happen if we dropped the minimum wage to 0 or took away all taxes) because there would be real impacts on people so it's unethical. Instead what economists do is look for places where there is an arbitrary difference between two economies. A good example is when one state raises the minimum wage and another state doesn't. You can now compare the economies between the two states to see whether he state with high minimum wage suddenly has inflation while the low minimum wage state doesn't. This would be an instrumental variable. This allows economists to look at natural experiments and derive causal results from these.

The issue with instrumental variables is that you can never be 100% certain that the instrumental variables are unrelated to the dependent variable. In the minimum wage example maybe the state that raised the minimum wage has strong labor unions which has some impact on inflation when the same effect wouldn't be true in the first state. It's the job of an economists to try and do their best to tease out these biases but in reality it's impossible to ever be 100% accurate.

So to answer your question "Why use econometrics instead of machine learning" the answer is that machine learning just can't be used most economic applications. It's an huge technology but is not a pancea and still can be wrong. There a great video series which explains what machine learning is and how it works that I'd recommend you watch to understand the subject deeper.

Answer 4

Here is a basic answer for anyone too uninterested to read the long answers:

1) ML focuses on prediction and not on causality (as does metrics) 2) ML is powerful for parameter selection and model validation 3) Many ML algs. are incredibly similar to basic metrics approaches. For example, ridge regressions and LASO are both just small extensions of OLS.

Answer 5

I think the essence of this question is actually asking the difference between statistics and econometrics. You can find some good answers here. Here is my try on a simple - and maybe abstract, but I think useful way of - classification of these things.

ML are statistics models. Econometrics are often the combinations of economics models and statistics models.

Statistics models only tell about the associations between data and data themselves contain no information about causality. One can never only use data to tell something causal.

Scientists or social scientists try to understand the world by building science or social science models. Science and social science models are based on assumptions, which are simplifications of the real world and further based on our accumulated knowledge on the domains under study. Only through these models scientists or social scientists can give some causal statements about the world. And then scientists or social scientists want to test or verify or falsify these statements by applying the relevant statistics models on the real world data and checking if the associations found in the data are in accordance with the casual statements.

Accordingly, economists build economics models and want to test or verify the causal statements of these economics models through statistics models. The existed statistics models or approaches sometimes turn out to be not suitable for the economics models, so economists sometimes build their own statistics models. IV is such an example that economists, having their economics models in mind, build a new statistics model which is relevant to their economics models and can be used to test the statements of their economic models.

So there is no perfect IV. Every IV is based on the relevant economics model. And if an IV is not plausible, it means the economics model and the assumption behind it is not recognized as plausible by other economists given the accumulated domain knowledge.

So there is also no genuine or authentic externality. Even randomized experiments are not refugees. RCTs are implicitly science models. They have assumptions (you cannot control everything!) and they give causal statements which are tested by the relevant statistics models.

Finally, my understanding is that science is not about truth or true causality, but about models helping us understand the world. (There is a science-philosophy debate on this but I think it has gradually becomes the consensus among human scientists in the 21th century.) Rather than looking for true causality, science improves simply by inventing newer and more useful models that help us understand the world better.