Consider the two models $ (a) y = X\beta + u $ where $X$ is $n \times K$ and (b) $y = Z\gamma + \omega $ where $Z$ is $n \times r$. Under classical assumptions (and $Z$ and $X$ are non-stochastic) if model (a), that is $y = X\beta + u$ is the true model, show that $E(\sigma^{2}_{\omega}) \geq \sigma^2_u$ and explain the implication of your result.

I can do it in two cases when $r>k$ (overfitting) or when $r<k$ (underfitting).

Is there any short way of doing this, without cases (because for both overfitting and underfitting variance is biased)? So, what I am asking is that can they be clubbed so that result doesn't depend on relationship between $r$ and $k$?

  • $\begingroup$ If you are asking whether the variance of $w$ is expected to be greater than the variance of $u$, shouldn't you write $\mathrm E [\sigma^2_w] > \mathrm E[\sigma^2_u]$? $\endgroup$
    – ahorn
    Commented Aug 14, 2019 at 10:53
  • $\begingroup$ No, because a) is a true model (resembling population) and b) is misspecified model, direction of misspecification is not given. That's why I took cases. But since in both the cases expected variance will be more than true variance (biased estimator). So, my ques is can these be clubbed ie. Can we show this without considering whether r is bigger or smaller to K. If so, how can we do it? $\endgroup$ Commented Aug 14, 2019 at 11:52
  • $\begingroup$ E. Sommer understood what I was asking. They made the appropriate edit to the notation for that inequality. $\endgroup$
    – ahorn
    Commented Aug 14, 2019 at 12:46
  • $\begingroup$ What do you mean by the "variance is biased"? If you estimate $\sigma^2$ by $s^2$ then the last random variable is unbiased for $\sigma^2$. $\endgroup$
    – Bertrand
    Commented Aug 14, 2019 at 16:15

2 Answers 2


From a) we obtain $y - \hat{y}_x = M_xy = M_xu$ and from b) $y - \hat{y}_z = M_zy = M_zX\beta + M_zu$, where $M_x$ and $M_z$ are the residual-maker matrices (the ones from the accepted answer of the question that you linked) and $\hat{y}_x$ and $\hat{y}_z$ are the predicted values of y based on the who linear models.

The estimate of the variance is $s_x^2 = \frac{(y - \hat{y}_x)^T(y - \hat{y}_x)}{n-k}$ for the first model and $s_z^2 = \frac{(y - \hat{y}_z)^T(y - \hat{y}_z)}{n-r}$ for the second.

Thus $E(s_z^2) = \frac{E[(M_zX\beta + M_zu)^T(M_zX\beta + M_zu)]}{n-r} = \frac{1}{n-r}( E[(M_zX\beta)^T(M_zX\beta)] + E[(M_zX\beta)^T(M_zu)] + E[(M_zu)^T(M_zX\beta)] + E[(M_zu)^T(M_zu)]) = \frac{1}{n-r}( E[(M_zX\beta)^T(M_zX\beta)] + 2E[(M_zX\beta)^T(M_zu)] + E[(M_zu)^T(M_zu)]) = \frac{1}{n-r}( E[||M_zX\beta||_2^2] + 2E[\beta^TX^TM_zu] + E[(M_zu)^T(M_zu)]),$

as $(M_zX\beta)^T(M_zu) = (M_zu)^T(M_zX\beta) = \beta^TX^TM_z^TM_zu = \beta^TX^TM_zu \hspace{.2cm} \in \mathbb{R}.$

Beacuse $Z, X -$ fixed $||M_zX\beta||_2^2$ and $\beta^TX^TM_z$ are constants.

Thus $$E(s_z^2) = \frac{1}{n-r}(||M_zX\beta||_2^2 + 2\beta^TX^TM_zE(u) + E(u^TM_zu)) =$$ $$= \frac{||M_zX\beta||_2^2}{n-r} + 0 + \frac{E(u^TM_zu)}{n-r} = \frac{||M_zX\beta||_2^2}{n-r} + \sigma^2 \neq \sigma^2 \hspace{.2cm} in \hspace{.1cm}general.$$


If the model $ (a) y = X\beta + u $ is true, then $\omega = u + X\beta - Z\gamma $ in $(b)$, and the claim follows.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.