# How much can we trust macroeconometric analysis?

I am a student of economics in my masters and I have learned quite a lot about microeconometrics (I mean mainly quasi-experimental methods / causality determination). Here my current understanding is that obviously you can't ever fully trust any analysis, but basically you can get to a point where results are relatively convincing, i.e. when one can make a strong argument for the identification assumption to hold (so parallel trends in a Dif-in-Dif, exclusion restriction in an IV approach, etc.). Alternatively, you get identification through a well designed RCT. So basically you can find ways to solve problems of reverse causality and omitted variables.

I have now recently started to learn more about macroeconometrics (i.e. time series methods) and it kind of appears to me that reverse causality and omitted variable bias kind of get ignored here (I am thinking of 3-4 variable VARs that obvisouly don't capture all the variables that might explain e.g. output or inflation). Is there some mathematical argument that makes these problems disappear magically? Or is macroeconometrics in general less convincing?

I have heard of the credibility revolution in empirical micro and that nothing similar happened in macro, but how much can we trust results from something like an IRF estimation or how convincing are arguments based on IRF results?

PS. I know that in forecasting you can be kind of agnostic to any causality issues, so in this question I am not asking about forecasts. Also I am not putting microeconomics vs macroeconomics in this question. I am only talking about two different econometric approaches.

• Most time series analysis depends upon an assumption that statistical properties of tomorrow are like the statistical properties of today, regardless of how parsimonious or how rich your model is in terms of parameters. That doesn't make me a sceptic on econometrics, but instead, it is important to understand it's limitations. Nov 18 '20 at 12:09

I do not think that macroeconomic analysis would be 'less trust worthy'. In the same ways as you can solve the issue of omitted variables and reverse causality in microeconomics you can do it macroeconomics as well. Macroeconomics, is not just vector autoregressions (VAR - which by the way corrects for endogeneity). Simultaneous equation techniques, even difference-in-difference (DiD) and regression discontinuity (RD) etc are used even in macro literature even if they are less common (see this recent paper on identification in macroeconomics Nakamura & Steinsson, 2018).

In addition I think you are actually exaggerating quality of quasi-experimental methods such as IV or DiD. For example, this recent paper shows that large portion of IV papers uses weak instruments as the rule of thumb that $$F$$-stat. should be above 10 is invalid and true threshold is around $$F>105$$ (see here Lee McCrary Moreira & Porter 2020). In addintion DiD are notorious for $$p$$-hacking and other issues, Duflo & Mullainathan (2004) literally have paper titled: "How much should we trust differences-in-differences estimates?" Spoiler alert their answer is not as much as people usually do.

I think it would be fair to say that a reasonable person could defend saying that typical VAR has probably on average less issues that let's say DiD (but to be clear I am not claiming that as even this lacks nuance as the model selection and specification is always case dependent there is no one model uber ales - even RCTs have a problem when it comes to generalization of results).

When it comes to just empirics in general I could not find any paper that would show that empirical research in macroeconomics is less replicable than in microeconomics. I would also take an issue with claim that macroeconomic models cannot control for all relevant variables. There are macro models (e.g. dynamic stochastic general equilibrium (DSGE)) that include scores of different equations and even more variables. Next, even despite of this when it comes to omitted variable bias you might have a valid point although a lot of attention is being paid to the issue it might sometimes be hard to solve, but with the reverse causality your accusation that macroeconomics does not pay enough attention is unfounded. Most published research in macroeconomics pays very large attention to reverse causality (especially in present day, maybe 50-60 years ago that would be valid point as well). In fact that is why models and estimation techniques like VAR, DSGE, IV, GMM and many other simultaneous equations approaches are so popular and dominate macroeconomics, so this second charge is simply unfounded.

However, when it comes to forecasts I would agree that macroeconomic forecasting is for sure less accurate then microeconomic forecasting (e.g. forecasting aggregate output vs firm forecasting demand for its onw products). However, there the fault would not be in the models themselves necessary but rather with the fact that macroeconomic aggregates contain more noise, are harder to measure, and are reported on lower frequency and of course with less data it is harder to build good forecasting model.

Lastly, it is not really easy to qualify how much you should trust a model in any field. You should approach any model and set of results with healthy doze of skepticism, be they from VAR or RTC or any other approach. Different situations will require different identification strategies. Furthermore, you should not put all faith into single paper but wait for replications/put the paper into larger context of pre-existing literature. I also tried to find some work that would compare replicability of microeconomics vs macroeconomics, but unfortunately there does not seem to be such paper but I would not expect macroeconomics having much worse replicability rates then microeconomics per se.

• I love reading your answers and comments because you know more about econometrics than I'll ever know and they're chock full of wisdom. Then again, that's not a huge compliment because I know very little :) . But why is your focus on reproducibility ? Something can be reproduced and still not be high quality. (i.e., bogus or useless ). Shouldn't the focus, in the end, be forecasting ability ? Nov 4 '20 at 15:17
• @markleeds thanks for the compliments, you make me blush :). Now to respond to the comment: when we talk about reproducibility we do not mean just take the same data use the same method and get the same result - of course this can be always done even with p-hacked papers. Reproducibility of a result is often colloquially used even for a situation where someone tries to test the same relationship even with slightly different method the results should be similar, or trying to test the same relationship on different sample and see if results are reasonably robust.
– 1muflon1
Nov 4 '20 at 15:22
• @markleeds for example colloquially people often refer to Nuemark and Wascher study that tried to reproduce the minimum wage study of Card and Kruger, using completely different dataset as reproduction study - card and Kruger used self reported data while nuemark and wascher used hard payroll data. If two studies like this agree the result is considered to be reproducible and if not it would not be (at least colloquially in economics - I am not sure if maybe wider statistical literature on uses reproducibility also in more narrow sense.
– 1muflon1
Nov 4 '20 at 15:27
• Maybe I did not express this well, I am in no way putting micro over macro theory in this question. Also I know that one can use quasi experimental methods for macroeconomic issues. I am only talking about time series econometrics vs. the quasi experimental stuff. I also know the DiD paper, but I think since 2004 a lot has gotten better (of course nothing is ever good enough though). Lastly, I think the advantage of the quasi exp. methods is that weaknesses can be revealed quite easily, e.g. a good paper will show and discuss how weak/strong an instrument is.
– ArOk
Nov 4 '20 at 15:33
• @1muflon1: good point. I basically agree that the more evidence the stronger the evidence. Unfortunately, I don't know what the word is in statistics ? In fact, I'm not sure if there is one word but atleast I understand what your definition of reproducibility is. Unfortunately ( or fortunately ), due to R and github, reproducibility in statistical academics is getting much hype these days but it has the "duplicate the results" meaning that I mentioned earlier. I'll try to look around when I have some time but there may not be one ? All the best and I'm sorry that I can't help there. Nov 5 '20 at 16:48