2
$\begingroup$

I am struggling to understand why M is not null since: $$\mathbf M=I−X(X′X){^-}^1X′=I−XX{^-}^1X'{^-}^1X′=I-I=[0] $$ What's wrong with that reasonning?

$\endgroup$
  • 6
    $\begingroup$ Because $X$ need not be square matrix, and so $(X'X)^{-1}\ne X^{-1}X'^{-1}$. You can't invert a non-square matrix. $\endgroup$ – Herr K. Mar 25 '16 at 18:53
  • $\begingroup$ @Herr K. Thanks! Of course the number of parameters is not always the number of observations! I do not know how to mark your comment as the answer though... $\endgroup$ – jeake Mar 25 '16 at 18:58
  • $\begingroup$ You can't. People don't get points for good comments. They are just there to be helpful in simple cases that can be addressed quickly. Just give him a nice smile like this :) It's etiquette ;^))) $\endgroup$ – Kitsune Cavalry Mar 26 '16 at 8:41
  • 2
    $\begingroup$ @HerrK It is short, but you could write it up as an answer. (You have my vote...) $\endgroup$ – Giskard Apr 24 '16 at 20:43
3
$\begingroup$

The inverse of some matrix $X$, $X^{-1}$, is defined for square matrices only (i.e. when $X$ has the same number of rows and columns). In the typical econometric applications, the data matrix $X$ usually has far more rows (observations) than columns (regressors). Formally speaking, the matrix $X$ in your definition of $M$ has dimension $n\times k$ but $n\ne k$ (actually $n\gg k$), and so it's not a square matrix.

Therefore the second equality in your derivation is not valid; in particular, $(X'X)^{-1}\ne X^{-1}(X')^{-1}$, because $X^{-1}$ and $(X')^{-1}$ are not defined.

$\endgroup$
4
$\begingroup$

This is more a related thought than anything else, but I thought it might be useful for people looking at this question in the future. Herr K. should write a short answer and have it marked as accepted.

Let $X$ be an $n \times k$ matrix with linearly independent columns. Let $S \equiv \text{span}(X)$.

Regression can be thought of as the following problem: Given an n-dimensional vector $y \in \mathcal{R}^n$, find the vector in $\hat{y} \in S$ that is closest to $y$ -- i.e. $$\hat{y} = \arg\min_{z \in \mathcal{R}^n} ||y - z||$$ It can be shown that the solution to this problem is given by $$\hat{y} = Py = X (X' X)^{-1} X' y$$ where we refer to the matrix $P = X (X' X)^{-1} X'$ as the "projector matrix." Then the matrix $M = (I - P)$ allows you to get the residuals of your regression by $\hat{u} = (I - P)y = y - \hat{y}$. As you said, if $X$ is square then $P$ reduces to the identity and the residuals are 0. This is insightful for several reasons.

  • Geometrically, if $X$ is square and has $n$ linearly independent columns then $\text{span}(X)$ is all of $\mathcal{R}^n$. Then the projection problem reduces to picking the vector in $\hat{y} \in \mathcal{R}^n$ that is closest to some vector in $y \in \mathcal{R}^n$. Obviously, if your vector is coming from the span of the set, the minimum distance choice would be the vector itself. We see this because as you showed $P = I$ in this case. Then $M = (I - P) = 0$ which means you have 0 residuals and you have a perfect fit of your data.
  • Additionally, this is why empirical economists are careful about "throwing in the kitchen sink" in terms of how many variables are included. If you have as many linearly independent variables as observations then you can get a perfect match whether your exogenous variables are related to your endogenous variables or not.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.