How much can we trust macroeconometric analysis?

Question

I am a student of economics in my masters and I have learned quite a lot about microeconometrics (I mean mainly quasi-experimental methods / causality determination). Here my current understanding is that obviously you can't ever fully trust any analysis, but basically you can get to a point where results are relatively convincing, i.e. when one can make a strong argument for the identification assumption to hold (so parallel trends in a Dif-in-Dif, exclusion restriction in an IV approach, etc.). Alternatively, you get identification through a well designed RCT. So basically you can find ways to solve problems of reverse causality and omitted variables.

I have now recently started to learn more about macroeconometrics (i.e. time series methods) and it kind of appears to me that reverse causality and omitted variable bias kind of get ignored here (I am thinking of 3-4 variable VARs that obvisouly don't capture all the variables that might explain e.g. output or inflation). Is there some mathematical argument that makes these problems disappear magically? Or is macroeconometrics in general less convincing?

I have heard of the credibility revolution in empirical micro and that nothing similar happened in macro, but how much can we trust results from something like an IRF estimation or how convincing are arguments based on IRF results?

PS. I know that in forecasting you can be kind of agnostic to any causality issues, so in this question I am not asking about forecasts. Also I am not putting microeconomics vs macroeconomics in this question. I am only talking about two different econometric approaches.

Answer 1

I do not think that macroeconomic analysis would be 'less trust worthy'. In the same ways as you can solve the issue of omitted variables and reverse causality in microeconomics you can do it macroeconomics as well. Macroeconomics, is not just vector autoregressions (VAR - which by the way corrects for endogeneity). Simultaneous equation techniques, even difference-in-difference (DiD) and regression discontinuity (RD) etc are used even in macro literature even if they are less common (see this recent paper on identification in macroeconomics Nakamura & Steinsson, 2018).

In addition I think you are actually exaggerating quality of quasi-experimental methods such as IV or DiD. For example, this recent paper shows that large portion of IV papers uses weak instruments as the rule of thumb that $F$-stat. should be above 10 is invalid and true threshold is around $F>105$ (see here Lee McCrary Moreira & Porter 2020). In addintion DiD are notorious for $p$-hacking and other issues, Duflo & Mullainathan (2004) literally have paper titled: "How much should we trust differences-in-differences estimates?" Spoiler alert their answer is not as much as people usually do.

I think it would be fair to say that a reasonable person could defend saying that typical VAR has probably on average less issues that let's say DiD (but to be clear I am not claiming that as even this lacks nuance as the model selection and specification is always case dependent there is no one model uber ales - even RCTs have a problem when it comes to generalization of results).

When it comes to just empirics in general I could not find any paper that would show that empirical research in macroeconomics is less replicable than in microeconomics. I would also take an issue with claim that macroeconomic models cannot control for all relevant variables. There are macro models (e.g. dynamic stochastic general equilibrium (DSGE)) that include scores of different equations and even more variables. Next, even despite of this when it comes to omitted variable bias you might have a valid point although a lot of attention is being paid to the issue it might sometimes be hard to solve, but with the reverse causality your accusation that macroeconomics does not pay enough attention is unfounded. Most published research in macroeconomics pays very large attention to reverse causality (especially in present day, maybe 50-60 years ago that would be valid point as well). In fact that is why models and estimation techniques like VAR, DSGE, IV, GMM and many other simultaneous equations approaches are so popular and dominate macroeconomics, so this second charge is simply unfounded.

However, when it comes to forecasts I would agree that macroeconomic forecasting is for sure less accurate then microeconomic forecasting (e.g. forecasting aggregate output vs firm forecasting demand for its onw products). However, there the fault would not be in the models themselves necessary but rather with the fact that macroeconomic aggregates contain more noise, are harder to measure, and are reported on lower frequency and of course with less data it is harder to build good forecasting model.

Lastly, it is not really easy to qualify how much you should trust a model in any field. You should approach any model and set of results with healthy doze of skepticism, be they from VAR or RTC or any other approach. Different situations will require different identification strategies. Furthermore, you should not put all faith into single paper but wait for replications/put the paper into larger context of pre-existing literature. I also tried to find some work that would compare replicability of microeconomics vs macroeconomics, but unfortunately there does not seem to be such paper but I would not expect macroeconomics having much worse replicability rates then microeconomics per se.