What happens if you regress the residuals on the regressor itself? [closed]

Suppose e hat is the OLS residual of a regression of Y on X. If not we regress e hat on X, what would the OLS coefficient be?

• Can you please clarify the question? What do you mean by what happens? You will get regression of residuals on your independent variable… nothing else will happen your PC won’t explode
– 1muflon1
Oct 8 '21 at 6:43
• I have elaborated on the main question. Bruce Hansen's Econometrics has this exercise question that asks what will the OLS coefficient be when the residual is regressed on the regressor. I was myself wondering what will be special about it. Oct 8 '21 at 6:47
• @Jacob I recommend actually running these two regressions in a program. That should give you a hint. Rereading the book chapter will probably give you the answer. Oct 8 '21 at 7:40

The answer is trivially zero. The estimate from regressing residuals $$e$$ on regressor is $$(X'X)^{-1} X'e.$$ But $$X'e = 0 \in \mathbb{R}^p$$, where $$p$$ is the number of regressors (this is just, e.g. the FOC's defining the OLS $$\hat{\beta}$$). So $$(X'X)^{-1} X'e = 0$$.

This has nothing to do with the exogeneity assumption $$E[\epsilon_i X_i] = 0$$ (or $$E[\epsilon_i|X_i] = 0$$). You can always run the regression, and the OLS residuals will always have property, regardless of whether exogeneity is economically plausible.

To fix notation, $$y_i = x_i^T\beta + \epsilon_i$$. If the standard assumption on non-multicollinearity is made, then the residuals $$\hat\epsilon$$ regressed on $$x$$ is zero.

In matrix notation, a model with $$k$$ regressors, $$x=(x_1,\dots,x_k)$$, and $$n$$ observations, we have, $$\begin{eqnarray} Y = X \beta + \epsilon,&\text{ where }& Y=\begin{bmatrix} y_1\\ \vdots\\ y_n \end{bmatrix} & X=\begin{bmatrix} x_{11}&x_{21}&\cdots &x_{k1} \\ x_{12}&x_{22}&\cdots &x_{k2} \\ \vdots & \vdots & \ddots & \vdots\\ x_{1n}&x_{2n}&\cdots &x_{kn} \end{bmatrix}, & \beta = \begin{bmatrix} \beta_1\\ \vdots\\ \beta_k \end{bmatrix} \end{eqnarray}$$ (take the first column to be a column of 1's if there is constant term) and that the least squares estimator of $$\beta$$, is $$\begin{equation} \hat\beta = (X^T X)^{-1}X^T Y.\label{beta-hat}\tag{1} \end{equation}$$ (Non-multicolinearity guarantees that $$X^T X$$ is invertible.)

To use computational brute force without any insight, regress the residuals $$\hat\epsilon:=Y-X\hat\beta$$ on $$x$$ assuming a linear model $$\begin{equation} \hat\epsilon = x^T\gamma + \delta.\label{epsilon-model}\tag{2} \end{equation}$$ Then apply \ref{beta-hat}, \begin{align} \hat\gamma&= (X^T X)^{-1}X^T\hat\epsilon\\ &= (X^T X)^{-1}X^T(Y-X\hat\beta)\label{X^T e=0}\tag{3}\\ &= (X^T X)^{-1}X^T(Y-X(X^T X)^{-1}X^TY)\\ &= 0. \end{align}

**corrected with the comments below, noting that, assuming $$X^TX$$ is invertible, (\ref{X^T e=0})$$\iff X^T\hat\epsilon=0\iff(\ref{beta-hat})$$.

• Thank you so much. I was getting the OLS coefficient as zero but I thought I must be doing something wrong. The intuition really helps. Thank you! Oct 8 '21 at 15:03
• @Jacob The $\hat \epsilon$ should probably be a $\hat \gamma$: the estimate of the coefficient of $X$ for the equation $(2)$. The intuition is that the coefficient is zero as $X$ is orthogonal to $\epsilon = Y - X \beta$.
– tdm
Oct 8 '21 at 16:10
• Do you mean exogenous rather than endogenous? Oct 9 '21 at 6:40
• The question is about residuals ("e hat" as the question puts it), not the (unobserved) error term $\epsilon$. Regressing the residuals on regressors always gives trivially 0, just by property of the OLS procedure. It has nothing to do with the exogeneity assumption, which is a condition on the population/DGP. Oct 9 '21 at 16:36
• @Michael. You're absolutely right! I was trying to give some insight w/o resorting to geometry, but as turns out, wrong. Hope I have not confused the OP. Thanks for the clarification.
– p.co
Oct 10 '21 at 14:22