What happens if you regress the residuals on the regressor itself? [closed]

Question

Suppose e hat is the OLS residual of a regression of Y on X. If not we regress e hat on X, what would the OLS coefficient be?

Answer 1

The answer is trivially zero. The estimate from regressing residuals $e$ on regressor is $$ (X'X)^{-1} X'e. $$ But $X'e = 0 \in \mathbb{R}^p$, where $p$ is the number of regressors (this is just, e.g. the FOC's defining the OLS $\hat{\beta}$). So $(X'X)^{-1} X'e = 0$.

This has nothing to do with the exogeneity assumption $E[\epsilon_i X_i] = 0$ (or $E[\epsilon_i|X_i] = 0$). You can always run the regression, and the OLS residuals will always have property, regardless of whether exogeneity is economically plausible.

Answer 2

To fix notation, $y_i = x_i^T\beta + \epsilon_i$. If the standard assumption on non-multicollinearity is made, then the residuals $\hat\epsilon$ regressed on $x$ is zero.

In matrix notation, a model with $k$ regressors, $x=(x_1,\dots,x_k)$, and $n$ observations, we have, \begin{eqnarray} Y = X \beta + \epsilon,&\text{ where }& Y=\begin{bmatrix} y_1\\ \vdots\\ y_n \end{bmatrix} & X=\begin{bmatrix} x_{11}&x_{21}&\cdots &x_{k1} \\ x_{12}&x_{22}&\cdots &x_{k2} \\ \vdots & \vdots & \ddots & \vdots\\ x_{1n}&x_{2n}&\cdots &x_{kn} \end{bmatrix}, & \beta = \begin{bmatrix} \beta_1\\ \vdots\\ \beta_k \end{bmatrix} \end{eqnarray} (take the first column to be a column of 1's if there is constant term) and that the least squares estimator of $\beta$, is \begin{equation} \hat\beta = (X^T X)^{-1}X^T Y.\label{beta-hat}\tag{1} \end{equation} (Non-multicolinearity guarantees that $X^T X$ is invertible.)

To use computational brute force without any insight, regress the residuals $\hat\epsilon:=Y-X\hat\beta$ on $x$ assuming a linear model \begin{equation} \hat\epsilon = x^T\gamma + \delta.\label{epsilon-model}\tag{2} \end{equation} Then apply \ref{beta-hat}, \begin{align} \hat\gamma&= (X^T X)^{-1}X^T\hat\epsilon\\ &= (X^T X)^{-1}X^T(Y-X\hat\beta)\label{X^T e=0}\tag{3}\\ &= (X^T X)^{-1}X^T(Y-X(X^T X)^{-1}X^TY)\\ &= 0. \end{align}

**corrected with the comments below, noting that, assuming $X^TX$ is invertible, (\ref{X^T e=0})$\iff X^T\hat\epsilon=0\iff(\ref{beta-hat})$.