1
$\begingroup$

Considering a Difference-in-Differences equation with a dummy variable:

$$ y_i = \alpha + \beta D_i + Xit + \varepsilon_i. $$ While $$Xit$$ is a set of covariates. Today when I add more covariates, the p-value decrease. Is it normal?

And if it is a problem, what should I deal with that?

$\endgroup$
1
  • $\begingroup$ It can happen if $D$ and $X$ are not much correlated each other and X is important for explaining $y$, when X decreases the variance of the (remaining) error and the variability in D is not much affected by partialing out X, resulting in a reduced standard error. $\endgroup$
    – chan1142
    Sep 22 at 14:37
3
$\begingroup$

First of all, let me tell you that it is bad statistics to compare $p$-values across different specification. What you certainly should not do is to pick the specification based on the $p$-values that you obtain.

Having said this. Assume that the right specification is given by: $$ y_i = \alpha + \beta D_i + \gamma X_{i} + \varepsilon_i $$ Assume you wrongly estimate: $$ y_i = \alpha + \beta D_i + \delta_i $$ where now $\delta_i = \varepsilon_i + \gamma X_i$ captures both the random error $\varepsilon_i$ and the effect of $X_i$.

If you would estimate the second specification, you actually identify the following: $$ \begin{align*} \hat \beta &= \mathbb{E}(y_i|D_i = 1) - \mathbb{E}(y_i|D_i = 0),\\ &= \beta + \mathbb{E}(\delta_i|D_i = 1) - \mathbb{E}(\delta_i|D_i = 0),\\ &= \beta + \gamma\left[\mathbb{E}(X_i|D_i = 1) - \mathbb{E}(X_i|D_i = 0)\right] \end{align*} $$ Depending on the second term, $\hat \beta$ might be lower or higher than $\beta$. This will depend on the sign of $\gamma$ and the direction of correlation between $X_i$ and $D_i$. For example if $\gamma > 0$ and $D_i = 1$ is associated with higher values of $X_i$ then $\hat \beta > \beta$, so you will tend to overestimate the effect of $D_i$ on $y_i$ (as you are also capturing part of the effect of $X_i$ in your estimate).

Whether you should include $X_i$ or not into the regression is not something that can be answered using statistics alone. If $X_i$ captures confounders (i.e. something that affects both $y_i$ and $D_i$), then yes you should add it as you otherwise do not capture the causal effect of $D_i$ on $y_i$ but also part of the effect of $X_i$ on $y_i$.

$\endgroup$
2
$\begingroup$

When you add covariates, $\beta$ is interpreted as "the effect of $D$ on $y$ holding fixed $X$. Adding variables to $X$ can either increase or decrease the p-value depending on the relationship of $X, D,$ and $y$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.