4
$\begingroup$

I have several macroeconomic models (eg two DSGEs and a VAR). They each produce forecasts for GDP, inflation and unemployment, and I have data to test them on, going back years.

How can I say which model has the "best" overall performance?

I realise this is vague, so here's more detail:

Assume that model X performs better than model Y on GDP, but worse on inflation.

If I'm just looking at GDP, I could just ask something like "which model has the smallest mean square error?" (predicted GDP - actual GDP, squared) Similarly, if I only care about inflation, I could do the same thing.

But I actually care about forecast accuracy over all three variables (GDP/inf/unemployment).

Is there a way to assess accuracy over all three variables, without just assigning weights to each MSE value and adding them up?

Thanks for your help!

$\endgroup$
2
$\begingroup$

The MSE is essentially a squared Euclidean distance between two vectors, say $\mathbf y$ and $\hat{\mathbf y}$, where $\mathbf y$ is the actual economic data over $T$ periods and $\hat{\mathbf{y}}$ the predicted values. A natural extension of this to matrices $\mathbf Y=(y_{it})$ and $\widehat{\mathbf Y}=(\hat y_{it})$ where $i=1,\dots,n$ and $t=1,\dots,T$ ($n$ variables over $T$ periods) would be \begin{equation} \text{MSE}^*=\frac{1}{nT}\sum_{i=1}^n\sum_{t=1}^T(y_{it}-\hat y_{it})^2. \end{equation}

But beware of the pitfall. Of course there are other matrix norms available as well.

$\endgroup$
2
  • $\begingroup$ Thanks for multivariate MSE info. However, the "units" are different for each dimension (eg GDP in dollars, inflation in %, unemployment could be % or absolute). Doesn't that mean I need to somehow "normalise" the values first, to make MSE a good measure? A simple example: if I measure GDP in $, model Y (great at inflation, ok at GDP) might have a lower overall MSE than model X (slightly better at GDP, much worse on inflation). But if I measure GDP in cents (or Yen, if in Japan), any gap in GDP becomes dominant, and model X suddenly looks way better than Y. Isn't this a big issue? $\endgroup$ – Mich55 Mar 26 at 4:19
  • 1
    $\begingroup$ @Mich55: Unit differences are indeed an issue that I overlooked. Normalization would seem sensible but it's not obvious how it should be done. A search in the model evaluation literature (e.g. this paper) may be helpful. $\endgroup$ – Herr K. Mar 26 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.