Why significant variables become insignificant in subsamples?

Question

I built an ordered logit using a sample of more than 4000 observations and the variable of the interest was significant. But then I split the sample into two subsamples: women and men and the variable became insignificant in both. How can I explain it?

Answer 1

This is because of several reasons. The standard logit test uses t-test to test whether coefficients are individually significant.

The t-test is given by:

$$t=\frac{\beta_i}{se_{\beta_i}}$$

where $\beta_i$ is the coefficient you estimated and $se_{\beta_i}$ is standard error of the estimated coefficient. In turn standard error is given by $se_{\beta_i}=\frac{s}{\sqrt{n}}$ where $s$ is the estimate of standard deviation and $n$ is the number of observations.

Lastly, to determine whether the variable is significant you compare whether the $t$ statistics is greater than critical value of the t-statistics ($t^*$).

The $t^*$ depends on several factors. First, it depends on the significance level you want to use (a commonly used level of significance will be 5% or $\alpha =0.05$. Second it depends whether you are using two-sided or one sided test, although in practice people would use almost always two sided test when running regression. Third, it depends on the degrees of freedom of the test which depend on number of observations ($df_t =n-1$). The magnitude of $t^*$ has inverse relationship with the number of observations $n$, that is higher $n$ lowers $t^*$ holding $\alpha$ and type of test constant.

Now given all the information above there are the following possible reasons why you found that the observations are no longer significant in the subsamples:

1. $\beta_i$ in a subsamples is lower than in the joint sample. Perhaps there are some outliers that have bigger impact on the coefficient size when you split the sample into sub-samples.

2. Standard error in the subsamples is higher. Since subsamples have lower number of observations and since standard error is given as $se_{\beta_i}=\frac{s}{\sqrt{n}}$ having lower number of observation, ceteris paribus, increases standard errors of your beta coefficients. Now the standard deviation $s$ itself can be different in a subsamples so it is not guaranteed the standard errors will be lower but they usually are.

3. The $t^*$ is higher in a smaller sampler since smaller samples have less degrees of freedom. For example, in a sample with 100 observations it is approximately 1.984 whereas with 1000 observations it would be approximately 1.962. Now you have quite a large number of observations and difference in $t^*$ in a sample with 2000 observations vs 4000 is very small, but there are certainly cases where that might be sufficient to make marginally significant coefficient insignificant.

4. Of course, combination of all of the above can be responsible.

To determine exact cause you can have a look at the $\beta$, $se$ and $t^*$ that are estimated, or in $t^*$ case used, in the regression you run. That will tell you exactly what combination of the above is responsible for coefficients becoming insignificant.