3
$\begingroup$

I am interested in the effect of an independent variable $x$ on a dependent variable $y$, like so

$$ y = \beta_0 + \beta_1 x + e $$

where $e$ is the error term. Now $x$ includes two effects $z_1$ and $z_2$. For simplicity, let say $x = z_1 + z_2$. If I include $z_1$ in the model, like this

$$ y = \beta_0 + \beta_1 x + \beta_2 z_1 + e$$

Does that mean that $\beta_1$ is predominantly capturing the effect of $z_2$?

And a follow up question is: if $z_1$ is correlated with the error term, will including it in the model bias the estimate of $\beta_1$?

$\endgroup$
8
$\begingroup$

If I include $z_1$ in the model, like this: $$ > y = \beta_0 + \beta_1 x + \beta_2 z_1 + e, > $$ Does that mean that $\beta_1$ is predominantly capturing the effect of $z_2$?

Yes. This can be seen using the Frish-Waugh-Lovell theorem:

If you regress: $$ y = \beta_0 + \beta_1 x + \beta_2 z_1 + e, $$ then $\beta_1$ will be the same as the corresponding coefficient of a modified regression: $$ \hat y = \gamma_0 + \beta_1 \hat x + \hat e \tag{1} $$ where $\hat x$ is the residual from regressing $x$ on $z_1$ and the same for $\hat y$. Now if we regress $x$ on $z_1$ then the residual is equal to: $$ M_{z_1} x, $$ where $M_{z_1} = 1 - z_1(z_1'z_1)^{-1}z_1'$ is the annihilator matrix. If $x = z_1 + z_2$ then: $$ M_{z_1} x = (1 - z_1(z_1'z_1)z_1')(z_1 + z_2) = M_{z_1}z_2 $$ As such, substituting in $(1$), we have: $$ \hat y = \gamma_0 + \beta_1 \hat z_2 + \hat e, $$ where $\hat z_2$ is now the residual from regressing $z_2$ on $z_1$. Using the Frish-Waugh-Lovell theorem, in the reverse direction, this gives that $\beta_1$ is also equal to the coefficient in the following regression: $$ y = \delta_0 + \beta_1 z_2 + \delta_2 z_1 + \varepsilon. $$ In other words, $\beta_1$ will also be equal to the coefficient for $z_2$ for a regression of $y$ on both $z_2$ and $z_1$. Notice, however, that in general $\beta_2 \ne \delta_2$ (so the coefficients for $z_1$ will not be equal in the two regressions).

Another way to see this is by immediately substituting $x = z_1 + z_2$ into the regression $(1)$ then: $$ \begin{align*} y &= \beta_0 + \beta_1(z_1 + z_2) + \beta_2 z_1 + e,\\ &= \beta_0 + \beta_1 z_2 + (\beta_1 + \beta_2) z_1 + e. \end{align*} $$ So the coefficient on $z_2$ in the new regression is identical to the coefficient on $x$ in the original one $(\beta_1)$ , while the coefficient on $z_1$ in the new regression is the sum of the coefficients on $x$ and $z_1$ in the original regression $(\beta_1 + \beta_2)$.

And a follow up question is: if $z_1$ is correlated with the error term, will including it in the model bias the estimate of $\beta_1$?

The opposite is true. Including $z_1$ into the regression will make the estimate of $\beta_1$ unbiased. Consider the following data generating process: $$ y = \beta_0 + \beta_1 x + e, \tag{2} $$ and assume that $e$ is correlated with $z_1$. Then we can write: $$ e = \gamma z_1 + \varepsilon. $$ where $\varepsilon$ is now uncorrelated with $z_1$. (and where $\gamma = \mathbb{E}(e z_1)/\mathbb{E}((z_1)^2) \ne 0$. Assume for simplicity that $e$ is uncorrelated with $z_2$.

Then the estimate of $\beta_1$ will be biased as the orthogonality condition $\mathbb{E}(e x) = 0$ is not satisfied. Indeed: $$ \begin{align*} \mathbb{E}(ex) &= \mathbb{E}(e z_2) + \mathbb{E}(\gamma z_1 z_1) + \mathbb{E}(\varepsilon z_1),\\ &= \mathbb{E}(\gamma (z_1)^2) \ne 0 \end{align*} $$ If we include $z_1$ into the regression. Then substituting $e = \gamma z_1 + \varepsilon$, into $(2)$ we can write: $$ y = \beta_0 + \beta_1 x + \gamma z_1 + \varepsilon. $$ And $\mathbb{E}(\varepsilon) = \mathbb{E}(\varepsilon x) = \mathbb{E}(\varepsilon z_1) = 0$. So by including $z_1$ into the regression, we can guarantee the residual $\varepsilon$ to be uncorrelated with all covariates. This means that $\beta_1$ is identified and its estimate will be unbiased.

$\endgroup$
3
  • $\begingroup$ Regarding Another way to see this..., if I simulate the data according to $y=\beta_0+\beta_1x+e$ with $x=z_1+z_2$ and then regress $y$ on $x$ and $z_1$, the estimated coefficient on $z_1$ is around zero and insignificant -- and far from $\beta_1+\beta_2=\beta_1+0=\beta_1$. Is that in line with what you are saying? $\endgroup$ Jun 29 at 12:00
  • $\begingroup$ @Richard Hardy What I am saying is that if you first regress $y$ on $x$ and $z_1$ and then $y$ on $z_2$ and $z_1$, then the coefficient on $z_1$ for the second regression is the sum of the coefficients for $x$ and $z_1$ of the first regression. In addition, the coefficient on $x$ in the first regression should be the same as the coefficient on $z_2$ in the second regression. In your case the coefficient on $z_1$ is zero as it is zero in your data generating process. $\endgroup$
    – tdm
    Jun 29 at 12:53
  • $\begingroup$ OK, thanks! I think my case is the same as OP's case, as I am following the setup in there. But even if that is not the case, you explanation is helpful. $\endgroup$ Jun 29 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.