# When should we exclude an independent variable out of the regression?

In my case, I have two questions that I want to ask here relating to correlations among variables:

1. Whether it makes sense if we test the impact of laws on two outcome variables that are correlated to each other (around 0.6)

2. I have a table of correlation of some independent variables as below

Note: The variables names on the first row are equal to those on the column, for example (workpl~s=workplaces, sorry for the inconvenience due to the character limitation in STATA)

I am wondering which variables should be excluded from the regression? Is it okay to retain all of them in the regression. From my understanding, between two variables having high correlation, we should exclude one. High correlation is defined by having a correlation value higher than 0.5 and lower than 1.

Comparing simple or multiple regressions estimates is in itself interesting. There are only two special cases where simple regression of $$y$$ on $$x_1$$ will produce the same OLS estimate on $$x_1$$ as regressing $$y$$ on $$x_1$$ and $$x_2$$. Let's see why.

$$\tilde{y}=\tilde{\beta_o}+\tilde{\beta_1}x_1$$ and the multiple regression analog $$\hat{y}=\hat{\beta_o}+\hat{\beta_1}x_1+\hat{\beta_2}x_2$$. There is the following relationship between $$\tilde {\beta_1}$$ and $$\hat{\beta_1}$$: $$\tilde{\beta_1}=\hat{\beta_1}+\hat{\beta_2}\tilde{\phi_1}$$ where $$\tilde{\phi_1}$$ is the slope coefficient of the simple regression of $$x_{i2}$$ on $$x_{i1}$$, $$i=1,...n$$. Therefore, $$\tilde{\beta_1}$$ differs from the partial effect of $$x_1$$ on $$\hat{y}$$. The confounding term is the partial effect of $$x_2$$ on $$\hat{y}$$ times the slope in the sample regression of $$x_2$$ on $$x_1$$.

There are two distinct cases where they are equal:

• the partial effect of $$x_2$$ on $$\hat{y}$$ is zero in the sample ($$\hat{\beta_2}=0$$)
• $$x_1$$ and $$x_2$$ are uncorrelated in the sample ($$\tilde{\phi_1}=0$$)

Showing this omitted variable bias in general requires a bit of matrix algebra and is not important here. All I want to show is that if you assume that both play a role, leaving either out, will lead in biased estimates. That is why defining an appropriate model is actually quite difficult. Not because of causality (alone) but really because correlation alone is not what matters in regression analysis. The simple regression result of $$\frac{sample \ covariance \ of \ x \ and \ y}{sample \ variance \ of \ x}$$ only works if the two conditions above are fulfilled. Otherwise, your estimator is biased.

What you want to address is multicollinearity. The wikipedia article also offers guidance for detection as well as it's implication.

I have never modelled anything related to laws but posing a question and answering it with statistics is a delicate and complex task. I usually follow something along the line of:

• What do I try to achieve?

• What is my hypothesis? What is the (economic) theory behind it?

• Has this been asked before somewhere? If so, what did they use and why?

• What kind of model will I need to use? GLM (OLS), ML, ARIMA,... and what functional form is best suited for this.

• What data will be needed to answer this (and satisfy the assumptions of the model of choice)

• How do I need to clean, transform and check my data before I can use it. Is it stationary? Is it noisy? Any structural breaks (Paul Volker, Great Moderation, Dot.com bubble, Subprime crisis, Covid crises, in your case new laws, to name a few)? How do I account for these regime switches?

• Is there a different factor influencing this.

• Am I at risk of omitted variable bias? Or multicollinearity?

• What tests are best suited to check for the problems above? How to check for stationarity, collinearity, ...

• Once a model is setup, data is collected and appropriately transformed, you check your results. For example, is my error term uncorrelated with me explanatory variable(s). If not, what did I miss. Redo the above.

• Am I overdoing it now (data mining)?

• How do the results compare to existing findings? How do I interpret them?

A correlation of 0.9728 is not something I think should be used.

• "Not because of causality (alone) but really because correlation alone is not what matters in regression analysis". Do you mean not causality or correlation is what matter, but we need to concern and weight both ? Sep 7, 2021 at 21:58
• Yes, both, but also if the relationship is actually linear. Look up "anscombe quartet" to see how misleading correlation can be. Sep 7, 2021 at 22:14