Subsampling is normally examined in many international studies (grouped by market developement, country governance,etc), @1muflon1 has a great explanation here in one case.

In his answer, there is one thing I am intersted in that there seems not enough space in the answer part so I raise a separate answer.

In his answer, he said

"Unless you have some specific reason to subsample (e.g. you have long panel an you believe the error structure is different between developed countries so you need special specification for either subsample) you should not do subsampling."

I am wondering what does "error structure" here mean. Doe this statement mean that, we should not subsample the whole sample to developed and developing separately and run the main equation for these both subsamples and then comparing the coefficients between them because the different in number of observations will swallow the significant level and deter us from making conclusion?


What is Error Structure

A good explanation on what error structure is, is already provided in this Cross-Validated answer. In short:

error structure in this respect is referring to the "element of randomness" in your model. For example, in least squares regression, we often assume that the error term of the model (i.e. residuals) follows a normal distribution

Error structure is basically the catch-all term on all behavior of the error term.

Why Subsampling is Problematic

Now to answer the rest of the question, lets use some arbitrary example where you have sample of both developed and developing countries.

If you assume that both developed and developing countries have the same error structure it does not make much sense to subsample. For example, to make everything simple lets assume sample where we have 10 developed and 20 developing countries with 15 years worth of observations each (meaning we have 450 observations) and if your model looks like this:

$$y_{it} = \alpha_i + \beta_1 T_{it} + \beta_2 X_{it} + \epsilon_{it} \tag{1}$$

where $y$ is outcome, $T$ some treatment, $X$ control, and $\epsilon$ error term.

Now, if we assume error term has the same overall structure it makes sense to examine the extra effect on developed countries in the full sample with interaction variable between treatment and developed country dummy ($T \cdot D$).

$$y_{it} = \alpha_i + \beta_1 T_{it} + \beta_2 X_{it} + \beta_3 T_{it} \cdot D_i + \epsilon_{it} \tag{2}$$

This will tell us exactly how the effect between developed and developing countries differ, since $\beta_3$ is the extra different effect for developed country, and $\beta_1 + \beta_3$ gives us the total effect in developed countries, and we got all this by simply sacrificing 1 degree of freedom for this one extra coefficient while our sample size is still 450. It is important to understand that power of many statistical tests (such as the $t$-test applied to $\beta$ coefficients) increases in the sample size and decreases when we have more parameters - see discussion of this in Stock and Watson Introduction to Econometrics Ch 3 ). This is because generally speaking the more degrees of freedom you have, the better the power of statistical test will be. For classical a $t$-test degrees of freedom are given by $n-k$ where $n$ is the number of observations, $k$ number of regressors. So in the above example (1) we have $n-k= 450- 32 = 418$ df (32 because we have 30 dummies for each country and 2 because of 2 other regressors), in the example (2) we would loose only 1 extra df due to interaction term so we would have: $n-k=450-33=417$df.

If we would subset the sample we would not get any extra information, and loose more dfs. In this situation we would have separate model for developed countries:

$$y_{it} = \omega_i + \gamma_1 T_{it} + \gamma_2 X_{it} + e_{it} \tag{3}$$

with 150 observations and 12 parameters (we have only 10 country dummies, so we have $150-12= 138$df in model 3, and for developing countries we have

$$y_{it} = \kappa_i + \lambda_1 T_{it} + \lambda_2 X_{it} + u_{it} \tag{4}$$

with 200 observations and 22 parameters we have $200-22=178$df. So by sub-setting the sample we are not gaining any new information since asymptotically $\gamma_1 = \beta_1 + \beta_3$ and $\lambda_1 = \beta_1$ from the model (2), but our point estimates of $\gamma_1$ and $\lambda_1$ are estimated with less precession, there is higher chance we will not be able to reject null even if alternative hypothesis is true. In addition, to compare the difference between developed and developing countries $\gamma_1-\lambda_1$ we would have to set up additional test, and in the model (2) we already get this directly from the coefficient $\beta_3$ since asymptotically $\beta_3 = \gamma_1-\lambda_1$ (so actually subsampling is even extra work, provided you know how to set up interaction dummies).

However, this is not to say you should never subsample. For example, if the error structure is different, for example, if the developed countries sample suffers from some endogeneity, or if it requires more controls than the developing sample because of omitted variable bias (which affects structure of error term), and if there is no good way to adjust for this problem in whole sample, creating subsamples might be better. If you can justify subsampling then you can do that, but if you just want to see how some variable affects different categories differently, then you should not automatically default to subsampling unless you can justify it by some further arguments.

  • 1
    $\begingroup$ Hi @1muflon1, I am curious why we lose "simply sacrificing 1 degree of freedom" here although I understand what is degree of freedom $\endgroup$ – BeautifulMindset Jul 23 at 12:02
  • 2
    $\begingroup$ @BeautifulMindset because we are adding 1 extra beta coefficient which means that our system has now one less degree of freedom during the estimation procedure. $\endgroup$ – 1muflon1 Jul 23 at 12:04
  • $\begingroup$ (2) "power of many statistical tests decreases when we have more parameters". Do you mean parameters here are betas ? So, the more beta, the less degree of freedom (df) is what you mean here? $\endgroup$ – BeautifulMindset Jul 23 at 12:04
  • 1
    $\begingroup$ @BeautifulMindset 1. yes of course betas are the model parameters by definition of the word parameter. Yes more beta parameters/coefficients (whichever terminology you prefer) the less degrees of freedom the parametric models (such as various regression models) have 2. Precision in statistics refers to the size of standard errors (from which you calculate confidence intervals) if with 95\% confidence $\beta_2 = 2 \pm 10 $ then thats very imprecise estimate as even though point estimate is 2 with 95\% confidence it could be anything between -8 and 12, you typically want as big precision as $\endgroup$ – 1muflon1 Jul 23 at 12:29
  • 1
    $\begingroup$ possible. For example, if with 95\% confidence we can say that $\beta_2=2±0.001$ then thats much more precise, since we know that with 95\% confindece the coefficient is somewhere between 1.999 and 2.001, which is obviously more precise than saying it can be anything from -8 to 12 $\endgroup$ – 1muflon1 Jul 23 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.