# How good is a forecast? (Many outputs)

I have several macroeconomic models (eg two DSGEs and a VAR). They each produce forecasts for GDP, inflation and unemployment, and I have data to test them on, going back years.

How can I say which model has the "best" overall performance?

I realise this is vague, so here's more detail:

Assume that model X performs better than model Y on GDP, but worse on inflation.

If I'm just looking at GDP, I could just ask something like "which model has the smallest mean square error?" (predicted GDP - actual GDP, squared) Similarly, if I only care about inflation, I could do the same thing.

But I actually care about forecast accuracy over all three variables (GDP/inf/unemployment).

Is there a way to assess accuracy over all three variables, without just assigning weights to each MSE value and adding them up?

The MSE is essentially a squared Euclidean distance between two vectors, say $$\mathbf y$$ and $$\hat{\mathbf y}$$, where $$\mathbf y$$ is the actual economic data over $$T$$ periods and $$\hat{\mathbf{y}}$$ the predicted values. A natural extension of this to matrices $$\mathbf Y=(y_{it})$$ and $$\widehat{\mathbf Y}=(\hat y_{it})$$ where $$i=1,\dots,n$$ and $$t=1,\dots,T$$ ($$n$$ variables over $$T$$ periods) would be $$$$\text{MSE}^*=\frac{1}{nT}\sum_{i=1}^n\sum_{t=1}^T(y_{it}-\hat y_{it})^2.$$$$

But beware of the pitfall. Of course there are other matrix norms available as well.

• Thanks for multivariate MSE info. However, the "units" are different for each dimension (eg GDP in dollars, inflation in %, unemployment could be % or absolute). Doesn't that mean I need to somehow "normalise" the values first, to make MSE a good measure? A simple example: if I measure GDP in \$, model Y (great at inflation, ok at GDP) might have a lower overall MSE than model X (slightly better at GDP, much worse on inflation). But if I measure GDP in cents (or Yen, if in Japan), any gap in GDP becomes dominant, and model X suddenly looks way better than Y. Isn't this a big issue? Mar 26, 2021 at 4:19
• @Mich55: Unit differences are indeed an issue that I overlooked. Normalization would seem sensible but it's not obvious how it should be done. A search in the model evaluation literature (e.g. this paper) may be helpful. Mar 26, 2021 at 17:07