2
$\begingroup$

We have the below expression: $$ P_{M_{1}.X_{2}} \cdot y $$ I guess its interpretation is that they are the fitted values obtained from the regression of $y$ on residuals that are obtained from the regression of $X_2$ on $X_1$.

But what do we really mean by this? what is the thing we do actually with that regressions?

and How we can show that $$ P_x - P_1 = P_{M_{1}.X_{2}} $$

$\endgroup$
3
  • 2
    $\begingroup$ What does $x$ denote in $P_x$? And what is $P_1$? Is your model linear? Please give more details... $\endgroup$
    – Bertrand
    Nov 30, 2022 at 21:26
  • $\begingroup$ $P_x$ vector of fitted values from the regression of $y = X_1 \cdot \beta_1 +X_2 \cdot \beta_2 + u$ and $P_1$ is the one for $y = X_1 \cdot \beta_1 + u_1$ $\endgroup$
    – Tatanik501
    Dec 1, 2022 at 6:45
  • 1
    $\begingroup$ @Tatanik501 Actually $P$ is the projection matrix which is a special case of a projection operator in the context of a finite dimensional subspace. $\endgroup$
    – easyliving
    Dec 9, 2022 at 8:43

1 Answer 1

3
$\begingroup$

As the above comment pointed out, it is always good to provide exact definitions when asking a mathematical question.

But since your notation (or the notation in the book/lecture notes you are reading) is fairly standard, I'll fill in the definition from my interpretation and then give an answer.

In a generic multivariate regression $y = X \beta + e$, we use $P_X$ to denote the projection matrix $P_X := X (X'X) ^{-1} X'$ and use $M_X := I - P_X$ to denote the annihilator matrix.

If we partition the big $X$ matrix into two parts as in $X = (X_1, X_2)$ and partition the big $\beta$ vector into $\beta = (\beta_1', \beta_2')'$ in the compatible way, then we can write $y = X_1 \beta_1 + X_2 \beta_2 + e$.

Theorem. The result we want to prove is that the projection matrix onto $X$ can be broken into two (orthogonal) parts: the projection matrix onto $X_1$ and the projection matrix onto $\tilde{X_2}$, where $\tilde{X_2} := M_{X_1} X_2 = (I - P_{X_1}) X_2$ is the residual from the projection of ${X_2}$ onto ${X_1}$.

Algebraically, what we want to show is $P_X = P_{X_1} + P_{\tilde{X_2}}$.

Before we prove this, we note that this result is a type of the so-called Frisch-Waugh-Lovell theorem in econometrics (https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93Lovell_theorem).

Proof. Using $X = (X_1, X_2)$, we expand $$ P_X = X (X'X) ^{-1} X' = (X_1, X_2) \left[\begin{pmatrix} X_1' \\ X_2' \end{pmatrix} (X_1, X_2)\right]^{-1} \begin{pmatrix} X_1' \\ X_2' \end{pmatrix} = (X_1, X_2) \begin{pmatrix} X_1'X_1 & X_1'X_2 \\ X_2'X_1 & X_2'X_2 \end{pmatrix}^{-1} \begin{pmatrix} X_1' \\ X_2' \end{pmatrix}. $$

We use the simplified notation $$ A := \begin{pmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{pmatrix} := \begin{pmatrix} X_1'X_1 & X_1'X_2 \\ X_2'X_1 & X_2'X_2 \end{pmatrix} $$ and also use $$ A^{-1} := \begin{pmatrix} A^{11} & A^{12} \\ A^{21} & A^{22} \end{pmatrix} $$ to denote the inverse of $A$.

Now, using bock matrix inverse formula (https://en.wikipedia.org/wiki/Block_matrix), we can derive $$ \begin{align} A^{22} & = A_{22.1}^{-1} := (A_{22} - A_{21} A_{11}^{-1} A_{12})^{-1} \\ & =(X_2'X_2 - X_2'X_1 (X_1'X_1)^{-1} X_1'X_2)^{-1} \\ & =(X_2' M_{X_1} X_2)^{-1} \\ & =(\tilde{X_2}' \tilde{X_2})^{-1},\\ & \\ A^{21} & = - A^{22} A_{21} A_{11}^{-1} \\ & = - (\tilde{X_2}' \tilde{X_2})^{-1} X_2'X_1 (X_1'X_1)^{-1},\\ & \\ A^{12} & = A^{21\prime}, \\ & \\ A^{11} & = A_{11}^{-1} + A_{11}^{-1} A_{12} A^{22} A_{21} A_{11}^{-1} \\ & = (X_1'X_1)^{-1} + (X_1'X_1)^{-1} X_1'X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2'X_1 (X_1'X_1)^{-1}. \end{align} $$

With these formulas, we go back to $P_X$ to derive $$ \begin{align} P_X & = (X_1, X_2) \begin{pmatrix} X_1'X_1 & X_1'X_2 \\ X_2'X_1 & X_2'X_2 \end{pmatrix}^{-1} \begin{pmatrix} X_1' \\ X_2' \end{pmatrix} \\ & = (X_1, X_2) A^{-1} \begin{pmatrix} X_1' \\ X_2' \end{pmatrix} \\ & = (X_1, X_2) \begin{pmatrix} A^{11} & A^{12} \\ A^{21} & A^{22} \end{pmatrix} \begin{pmatrix} X_1' \\ X_2' \end{pmatrix} \\ & = X_1 A^{11} X_1' + X_1 A^{12} X_2' + X_2 A^{21} X_1' + X_2 A^{22} X_2' \\ & = X_1 (X_1'X_1)^{-1} X_1' + X_1 (X_1'X_1)^{-1} X_1'X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2'X_1 (X_1'X_1)^{-1} X_1' \\ & - X_1 (X_1'X_1)^{-1} X_1'X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2' \\ & - X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2'X_1 (X_1'X_1)^{-1} X_1' \\ & + X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2' \\ & = P_{X_1} + P_{X_1} X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2' P_{X_1} \\ & - P_{X_1} X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2' - X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2' P_{X_1} + X_2 (\tilde{X_2}' \tilde{X_2})^{-1} X_2' \\ & = P_{X_1} + (X_2 - P_{X_1} X_2) (\tilde{X_2}' \tilde{X_2})^{-1} (X_2' - X_2' P_{X_1}) \\ & = P_{X_1} + \tilde{X_2} (\tilde{X_2}' \tilde{X_2})^{-1} \tilde{X_2}' \\ & = P_{X_1} + P_{\tilde{X_2}}. \\ \end{align} $$ Q.E.D.

P.S. Let me know if there is any typo. This is a lot of typing. :)

$\endgroup$
1
  • $\begingroup$ Thanks a lot for your detailed answer :) it really helps me to see full algebraical expression to understand what we really do. I tried to do something similar but I was lost. I appreciate your time. $\endgroup$
    – Tatanik501
    Dec 10, 2022 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.