# What should we do if the subsample have the opposite results to the general results?

In my replication study, I examine whether a law (a law implemented staggered by different countries) has an impact on firms' cash equivalents. The result turning out for the whole sample is that the firms' cash equivalents decrease after this law.

However, when I subsample the data by developed and developing countries, the result is funny. It shows that in developed countries, cash equivalents increase while in developing countries, cash equivalents decrease.

Did you face the same problem before, what do you normally do to solve this things. My senior friend told me it is a kind of paradox, it is better that we ignore this kind of subsample. Their argument is that the subsample should present the same direction with the whole sample.

I did control for group and period-fixed effects and some relevant covariates

I'd interact the regressor you are interested in with a dummy for the country being developed and see what happens. Its entirely possible that the mechanisms at play in developed contries are different from those in the rest of the world. Depending on what your goal is you might satisfy yourself with the observation that the effects are different for the two groups of countries or you might think about modelling the transmission mechanism explicitely.

• Is there any reference for this way of testing, may I ask ? Jul 1 at 8:35
• @BeautifulMindset, you have two dummies - legistation - {yes, no} and developed - {yes, no}, the product of the two dummies is also a dummy. Its one if both are yes. The value of the of the coefficient on that dummy is the difference between (yes, yes) and (legislation = yes, developed = no), as you do net include the developed dummy in he regression in its own. It is tested by just adding the interaction to the regression equation for exmaple in R you would add "+legislation:developed" to the formula object Jul 1 at 11:33

tldr: As the other two answers also indicated, there is not necessarily a problem with your results. It might be the case that the two subgroups have different distributions of the covariates. Alternatively, it might be the case that the within group effects of the law are different from the between group effect.

### Joint and separate regressions

Consider two groups indexed by 0 and 1. Assume that there are $$k$$ covariates. The regression on the full subsample can be written as: $$y = \underbrace{X}_{n \times k} \underbrace{\beta}_{k \times 1} + \varepsilon$$ The value of the estimator of $$\beta$$ is given by: $$\hat \beta = (X'X)^{-1} X'y$$ When considering the first subsample, we can condition on group $$0$$: $$y_0 = X_0\beta_0 + \varepsilon_0$$ The value of the estimator of $$\beta_0$$ is given by: $$\hat \beta_0 = (X_0'X_0)^{-1}X_0'y_0$$ Likewise, on the second subsample, we have: $$y_1 = X_1 \beta_1 + \varepsilon_1$$ and: $$\hat \beta_1 = (X_1'X_1)^{-1}X_1'y_1$$ The estimates $$\hat \beta, \hat \beta_0$$ and $$\hat \beta_1$$ are related in the following way, \begin{align*} \hat \beta &= (X'X)^{-1} X'y,\\ &= (X'X)^{-1} X_0'y_0 + (X'X))^{-1}X_1'y_1,\\ &= W_0 \hat \beta_0 + W_1 \hat \beta_1, \end{align*} where $$W_1 = \underbrace{(X'X)^{-1} X_0'X_0}_{k \times k},\\ W_2 = \underbrace{(X'X)^{-1} (X_1'X_1)}_{k \times k}.$$ One can check that $$X'X = X_0'X_0 + X_1'X_1$$ so we see that $$W_1 + W_2 = I$$. This means that the coefficients of the pooled regression are weighted sums of the coefficients of the sub-sample regressions. Notice, however that the weights should not necessarily be non-negative. Also, as $$W_1$$ and $$W_2$$ are of dimension $$k \times k$$, each coefficient in $$\hat \beta$$ is (potentially) a function of all the coefficients in $$\hat \beta_0$$ and $$\hat \beta_1$$.

One exception is when the distribution of $$X_0$$ and $$X_1$$ are identical. Intuitively, this corresponds to the notion that $$X$$ is distributed independent of group membership. In this case, $$X_0'X_0 \approx X_1'X_1$$, so $$W_1$$ and $$W_2$$ are diagonal matrices with on the diagonals the relative sample sizes of the two subgroups. In this case, every coefficient of $$\hat \beta$$ is then a weighted average of the corresponding coefficient in $$\hat \beta_1$$ and $$\hat \beta_2$$.

#### Intuition

To see the intuition behind above result, consider the picture below. There's 6 data points divided in two groups: the green and blue dots. The within group regressions (red lines) give a negative slope. However, if we run the regression on the entire sample, we get a positive slope (purple line). This is due tot the fact that the second group has both (on average) higher values of $$x$$ and higher values of $$y$$, that overturn the negative within group association. As an example consider the association between firm wages and employment and assume that you have a sample across different sectors. It is possible that within each sector there could be a negative association (as higher wages may lead to less profits). However, it might also be the case that across sectors, there is a positive relationship: as more profitable sectors pay higher wages.

In your case, you have group fixed effects for the joint regression. Putting joint fixed effects is the same as subtracting from both $$y$$ and $$L$$ (the law variable) the within group mean of these variables. As such, for these normalized variables, say $$\bar y$$ and $$\bar L$$, you have the joint regression: $$\bar y = \beta \bar L + \varepsilon$$ and the group specific regressions (without intercept as the variables are de-meaned): $$\bar y_0 = \beta_0 \bar L_0 + \varepsilon_0,\\ \bar y_1 = \beta_1 \bar L_1 + \varepsilon_1$$ The estimates are given by: \begin{align*} &\hat \beta = \frac{\bar L'\bar y}{\bar L'\bar L},\\ &\hat \beta_0 = \frac{\bar L_0' \bar y_0}{\bar L_0' \bar L_0},\\ &\hat \beta_1 = \frac{\bar L_1' \bar y_1}{\bar L_1' \bar L_1}. \end{align*} Then: \begin{align*} \hat \beta &= \frac{\bar L_0' \bar y_0 + \bar L_1'\bar y_1}{\bar L'\bar L},\\ &= \hat \beta_0 \underbrace{\frac{\bar L_0'\bar L_0}{\bar L' \bar L}}_{w_0} + \beta_1 \underbrace{\frac{\bar L_1' \bar L_1}{\bar L' \bar L}}_{w_1},\\ &= \hat \beta_0 \frac{n_0 p_0(1-p_0)}{n_0 p_0(1-p_0) + n_1 p_1(1-p_1)} + \hat \beta_1 \frac{n_1 p_1(1-p_1)}{n_0 p_0(1-p_0) + n_1 p_1(1-p_1)} \end{align*} where $$n_0$$ and $$n_1$$ are the subgroup sample sizes and $$p_0$$ and $$p_1$$ are the fraction of treated observations within each subgroup.

We see that both $$w_0$$ and $$w_1$$ are non-negative, so $$\hat \beta$$ is a weighted average of $$\hat \beta_0$$ and $$\hat \beta_1$$. However, it is still possible that one of the two is negative and the other is positive.

If both groups are equal size, then the group with the largest variance (in $$L$$) will have the largest weight. If both group have the same variance, then the group with the largest number of observations will have the largest weight.

• an amazing work for explaining, @tdm, I totally and really appreciated it, but I am looking for a solution. I understand more from your dedicated explanation. Many thanks and warm regards. Jul 1 at 21:50
• Hi @tdm, can I ask where $W_1 + W_2 = I$ coming from ? What does $I$ mean ? Jul 2 at 22:28
• And can I ask what do the "the fraction of treated observations within each subgroup" mean? Jul 2 at 22:54
• @BeautifulMindset Assume that there are $n_1$ observations in subgroup $1$ and $m_1$ have the law implemented (i.e. have the value $L = 1$). Then the fraction of treated observations in subgroup 1 is $\frac{m_1}{n_1}$
– tdm
Jul 3 at 12:39

This sounds like a case of Simpson's Paradox. Did you control for fixed effects? You might also have heterogeneity - there may be different results in developing vs developed countries.

Generally, it's not wise to ignore something interesting in your data, but I defer to the advisor who's familiar with the area. It may be a known phenomenon that isn't interesting to people in the domain anymore.

• @BeautifulMindset no this could be Simpsons paradox, here the additional control you omit would be dummy for being developing country, of course such dummy would clash with fixed effects, but as Grada recommended you can try to do treatment*developing dummy interaction that will work even with fixed effects, or you can just try to use random effects (if applicable) and then just add the dummy
– 1muflon1
Jul 1 at 7:55
• @1muflon1 , so do you mean, normally, my regression is : y= post * treat + firm fixed effect + year fixed effect + error term. Now, I will create a variable calling developed equalling to 1 this observation belongs to developed countries. And whether you mean, now, my regression equation is: y= post * treat * developed + firm fixed effect + year fixed effect + error term Jul 1 at 8:34
• if you have both post x treat and post x treat x developed then beta on post x treat gives you the result conditional on the fact that the treatment effect on developed is different, then to see what the difference is you just look at beta for post x treat x developed. Also what you say is equivalent to what I say, if developing countries exhibit opposite effect to developed then developed must by definition exhibit opposite effect to developing, so you can put either developed = 1 developing= 0 or developed =0 developing = 1, results wont qualitatively change, you should set baseline
– 1muflon1
Jul 1 at 21:48
• to what you believe is more general case by convention but other than that it does not matter what you set as your baseline
– 1muflon1
Jul 1 at 21:49
• @BeautifulMindset ?? Well your question here was: What should we do if the subsample have the opposite results to the general results? If subsamples have different result, then you should have additional dummies and interaction. For example, if education affects men and women differently then you should not run 2 separate regressions for men or women (except for some special cases), you should add an interaction educ*gender (and of course still have educ itself), allow for different effect between genders
– 1muflon1
Jul 1 at 21:56