3
$\begingroup$

In my understanding the utility of econometrics versus machine learning is generally argued to be the research of causality.

If I'm not wrong, when we put aside experimental economics, the golden standard to do prove causality is to use Intrumental Variables. But in practice it has been showed that finding a relevant IV is often not possible, and authors end up taking the best "IV" they can found.

Problem: This variable has a high chance to not in fact be exogenous, but we can not verify it when the IV is dependant of variables like happiness, for example. Since we have to no way do verify those dependances, science stops here.

SO I am wrong to say that in most cases econometrics can not scientifically establish causality? In that case why using traditional econometrics instead of modern machine learning techniques?

$\endgroup$
6
$\begingroup$

I.) 2 Principles of econometrics can potentially be useful compared to Machine Learning. (see Hal R Varian 2014 Paper : https://pubs.aeaweb.org/doi/pdf/10.1257/jep.28.2.3)

A.) As you suggest the search of causality is one advantage but unlike what you think, even if causality sometimes could be tricky to measure it remains very useful and functional.

But first, when you suggest that Instrumental variable is the only available tool which work for most of the cases and allow causal inference, i think there is a few more techniques that could still be applicable to measure causality in response to a treatment, a manipulation, or an intervention and still be relevant (unlike natural experiment as you precise because of its restricted direct applicability to most of the cases) in most of the situations such as:

● explicit experiments ● regression discontinuity ● difference in differences ● structural estimation

For instance you can investigate causality with theses techniques or instrumental variables because even if those techniques are subject to bias and correlation issues (all the more since big data because the increasing size of datasets limits the usefulness of instrumental variable methods, depending on instrument strength and level of confounding), it still give a hint to investigate causality. In a way that, for instance, it enables the detection of conditional counterfactual (an if-clause which is contrary to fact) while also check the existence of potential selection bias.

What is a Conditional counterfactual:

If it is raining, then he is inside. : Rain = Indicative Variable

If it were raining, then he would be inside. : Rain = Conditional Counterfactual variable

Hence, the better predictive model you have for the counterfactual, the better you will be able to estimate the causal effect. Thus, even though a predictive model will not necessarily allow one to conclude anything about causality by itself, such models may help in estimating the causal impact of an intervention when it occurs. Because it can highlights a bunch of conditional counterfactual which could be use as potential causal Variables to run the test on, in order to eventually determined causality in a given dataset to do good decision making (clean up from confounding issues). If not, at least, it gives you some clues to conduct a deeper investigation to understand the problem : anything wrong with the theory ? anything wrong with my econometric model ? anything wrong with my data ?.

For more insight about the pro cons of differents causal inferences techniques in econometrics :

https://www.jstor.org/stable/pdf/44234997.pdf?refreqid=excelsior%3A18bacfd86299dcf19e7f1f13d9c52022

B.) Except causality, model uncertainty is another advantage of Econometrics compared to ML.

Probabilistic foundation of econometrics is a strenght in a way that it allows interpretability of most of the models and their parameters (avoiding black box phenomenon) and give a quantisation of uncertainty (with confident intervals ) . The goal is usually to show that the estimate of some interesting parameter is not very sensitive to the exact specification used : how an estimated parameter varies as different models are used.This question illustrate a simple form of model uncertainty. In this period of “big data,” it seems strange to focus on sampling uncertainty, which tends to be small with large datasets, while completely ignoring model uncertainty, which may be quite large. One way to address this is to be explicit about examining how parameter estimates vary with respect to choices of control variables and instruments.

II.) On the contrary Machine Learning techniques could also be useful for data analytics in social Science.

A.) Parameters Selection, Model validation methods in ML can improve traditional econometrics models

Researchers in machine learning have developed ways to deal with large datasets and economists interested in dealing with such data would be well advised to invest in learning these techniques. For instance, Web Mining methods could discover new usuable explanatory variable. Cross validation should identify a non-linear effect or a forgotten cross effect. Model Validation should detect when a model is mispecified and thus allows a better specification of an econometric model and on the whole reduce omitted variable bias and error. As an example, recent litterature in finance focus on Garch models (traditional times series models) improved with Neural Network to better predict volatility and asset prices.

For more insight about the usefulness of ML in econometrics check out this paper by Arthur Charpentier : https://arxiv.org/pdf/1708.06992.pdf

B.) Causal Modelling began to be a concern and a field of research in ML. Which means that in the long run ML could potentially overcome its own flaws ( like being exclusively focus on the fit ) and outperform Econometrics.

Some Theoretical computer scientists, such as Pearl (2009a, b) have made significant contributions to causal modelling in computer science (by extension machine learning):

(see: Causal Inference in Statistics: A Primer Wiley, 2016 Judea Pearl et Al)

Pearl defines counterfactuals directly in terms of a "structural equation model" – a set of equations, in which each variable is assigned a value that is an explicit function of other variables in the system. Given such a model, the sentence "Y would be y had X been x" (formally, X = x > Y = y ) is defined as the assertion: If we replace the equation currently determining X with a constant X = x, and solve the set of equations for variable Y, the solution obtained will be Y = y. This definition has been shown to be compatible with the axioms of possible world semantics and forms the basis for causal inference in the natural and social sciences, since each structural equation in those domains corresponds to a familiar causal mechanism that can be meaningfully reasoned about by investigators.

However, it appears that these theoretical advances have not as yet been incorporated into machine learning practice. Except in some recent research paper :

http://www.nasonline.org/programs/sackler-colloquia/documents/athey.pdf

As a Conclusion, in my opinion and as things now stand, Econometrics as more valuable input in social science than what machine learning could bring. So it is still adequate to use econometrics to conduct data analysis instead of ML techniques.

$\endgroup$
1
$\begingroup$

My view coincides with the introduction to your question. Namely, a) Econometrics is mostly concerned with causality b) Machine learning is mostly concerned with fit

But for the remaining part, our views depart. Here is why:

a) IV (and other quasi-experimental techniques) are not the only way to test for causality. The alternatives are i) experiments ii) structural estimations. In both cases you apply econometric machinery though you will mostly use a simple OLS in the first case and bayesian/GMM/Maximum Likelihood things in the second. Compared to experiments, it is probably less clear how structural estimations help and here I come to the second point;

b) As in any science, economists build math models of how things work. The problem is, there are many models for many contexts. How to define, which one is an appropriate in given circumstances? Here is where the econometricians help, because econometrics helps to discriminate between working and non-working models. There are different ways to show it: with (the now so popular) quasi-experiments, you show that the (hopefully causal) link either exists or does not and its magnitude = $\beta_i$. Now, what if your data does not fit into the shoes of the approach and doing an experiment is impossible? You can go for structural estimation You say: "Hey! Let's assume that the model XYZ - e.g. the Cobb-Douglas production function - works. If this is true, what would be the estimated parameters?" So you take your non-experimental data and forcefully put them into the model you have and estimate the parameters. How does it help to establish the "truth"? You look at the parameters and try to understand how reasonable they are. For instance, if you investigated the parameters of the Cobb-Douglas production ($Y = AK^\alpha L^{1-\alpha}$ s.t. $\alpha < 1$) function but your coefficients for the log-log regressions are $\beta = [13, 0.8, 1.5]$ (the numbers are absolutely fictional) then you have a reason to conclude that the model does not fit the context (maybe the industry) you are studying because you got $\beta_2 = (1 - \alpha) = 1.5 > 0$. Nonsense, right? Maybe, because you can blame data, the fact that you observe the equilibrium outcomes and omitted variable bias. But it makes you think in both directions: anything wrong with the theory? anything wrong with my econometric model?

People in Macro and IO frequently follow the approach (their tools differ though) because one has limited abilities to experiment in the field. Otherwise, when you do follow the prescriptions of the quasi-experimental literature, you have investigate only a little subset of problems, which are crucial for our understanding of how economies work. This is in my opinion the main point of the Deaton's critique with respect to the quasi-experimental approach to the causal inference. It turns researchers into people, who are looking for a problem that fits the tool and care about the context in asmuch as randomization is credible without digging into the field for real. They can publish studies on labor economics, make conclusions about political economy, analyze sports data and evaluate development policies in poor countries all at the same time and do not care about the underlying mechanisms. The quasi-experimentalist don't care about the econometric model as long as randomization works. Maybe the true relationship is linear, maybe not. But when instruments are strong and you use the right words in the section on identification strategy, it does not matter for the publication? This is probably not that bad per se, but Deaton is worried that the approach tell little about which models work and what are the values of the fundamental parameters (Check the response of Imbens though. Both things are good readings).

Why are the parameters important? Let me give an analogy from physics (where my knowledge is limited by my high school classes). In physics you measure the things, make experiments and obtain the coefficients. Fine, now we want to make a prediction about how fast the stone falls in a new place. The natural appoach is to use the previously obtained parameter estimates to predict how fast a stone falls from a certain distance in a new environment. If you know how the new environment differs from the other one (defined by the model!) you can use the coefficients you got, plug them into a model and get a credible prediction.

In economics, the values you get from the quasi-experiments won't help you to do the same thing. Think of a development program the World Bank started in Eastern Europe and want to apply in South Africa. Assume you had a credible inference with a super-fancy RDD strategy. Fine, you got your super-significant $\beta_i$. It is clear though that the impact of development program in Eastern Europe won't be the same as in South Africa because the context (environment) differs. So using the $\beta_i$ directly won't work. But can't we just somehow adjust the values and make a reasonable prediction? Well, since we don't know what on Earth the $\beta_i$ really is and how two models really differ, we don't know what kind of transformation to the $\beta$ we need to apply. So we know something for Eastern Europe but cannot use the numbers for other places. That's a pity right? Cause you did a good job, but cannot generalize it's results. Structural econometrics can be explicit on what the coefficients - in terms of the model - mean and how to use the values when you transfer them to another environment. The price of it are stricter assumptions on the relationship between the variables and the structure you - as a modeller - impose on the error term.

c) In my opinion, machine learning is a valuable tool to gather the previously unavailable data and test things, which we could not test before. A good example is the current stream of papers on protests and political economy. With internet you get access to a lot of unstructured information. The ability to extract it using the machine learning techniques enables you, for instance, to evaluate the sentiments of the electorate and how it affects future political outcomes. So in a sense ML is a good tool to save your time to create datasets to study novel problems or tackle the omitted-variable bias (since now you can quantify things, which remained unmeasured - and staying part of an error term - before).

$\endgroup$
0
$\begingroup$

So you are right. It is extremely difficult to prove causality in economics. Using an instrumental variable is a good way to do so. I think you might be a little confused about the difference between "machine learning" and Econometrics.

Machine learning works in 2 ways:

1) You have a massive data set with the correct answers already keyed in. You split the data set in 2 into a test and train set, then run programs to create a function to identify the already correct answers given a variety of variables within the train set. You then can use the currently unseen test set to see how accurate the function that the computer has to see how accurate it is. A good example of this is computers figuring out which hand drawn number represents what real number because you can feed in pre-existing data sets.

2) You have no data but a fitness function (a function which defines correctness). You create your own data by having the computer randomly set the function that turns data into the output and then put the many random functions into a situation. After the simulation finishes then you see which functions performed the best and modify all the functions to be more like that. Over thousands or millions of iterations the functions slowly "tune" the functions so that the give the correct answer given the repeated situations. A good example of this would be self driving cars. Some researchers have literally loaded computers up into grand theft auto to allow it to practice.

Both cases have very limited applications to economics. In the first scenario you would need to have data sets on thousands of economies over thousands of years all with differing data. We certainly do have plenty of economic data but when you think about it we only have 195 countries to look at and there is only 1 global economy. We also only have around 100 years of good economic data. More over the relationships between variables seem to change. Conventional wisdom used to be the low unemployment led to inflation which doesn't seem to be the case anymore in the U.S. economy.

With the second machine learning practice there is no way to simulate the entire world economy accurately enough to test how different policies would impact the world economy due to not strong enough computer power.

Econometrics uses instruments because it allows for a natural experiment. You obviously can't just run experiments in the economy to see what happens (ex: What would happen if we dropped the minimum wage to 0 or took away all taxes) because there would be real impacts on people so it's unethical. Instead what economists do is look for places where there is an arbitrary difference between two economies. A good example is when one state raises the minimum wage and another state doesn't. You can now compare the economies between the two states to see whether he state with high minimum wage suddenly has inflation while the low minimum wage state doesn't. This would be an instrumental variable. This allows economists to look at natural experiments and derive causal results from these.

The issue with instrumental variables is that you can never be 100% certain that the instrumental variables are unrelated to the dependent variable. In the minimum wage example maybe the state that raised the minimum wage has strong labor unions which has some impact on inflation when the same effect wouldn't be true in the first state. It's the job of an economists to try and do their best to tease out these biases but in reality it's impossible to ever be 100% accurate.

So to answer your question "Why use econometrics instead of machine learning" the answer is that machine learning just can't be used most economic applications. It's an huge technology but is not a pancea and still can be wrong. There a great video series which explains what machine learning is and how it works that I'd recommend you watch to understand the subject deeper.

$\endgroup$
0
$\begingroup$

Here is a basic answer for anyone too uninterested to read the long answers:

1) ML focuses on prediction and not on causality (as does metrics) 2) ML is powerful for parameter selection and model validation 3) Many ML algs. are incredibly similar to basic metrics approaches. For example, ridge regressions and LASO are both just small extensions of OLS.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.