1
$\begingroup$

I have experimental data on students, retail investors, and professionals. The setup (experimental procedure is exactly identical). Each experiment consists of 5 rounds of decision-making.

However, my sample size of observations (participant multiplied by 5 rounds) is different in each experiment because of 1) a differing starting number of participants - for example, I could recruit almost 500 retail investors but only 100 students. 2) While the data cleaning rules are identical (where I drop observations), they impact all three experiments differently. For example, I am left with 480 retail investors but only 79 students.

Each experiment has its own Stata file and Stata do file.

Granted I run the same regression; here is my question:

Can I compare and contrast the results from the different groups while having different observations and different clusters (I cluster the standard errors at the individual level)?: e.g., the treatment condition is associated with a 10 percentage point increase in the dependent variable in professionals but with negative 5 percentage points in retail investors. In contrast to investors and professionals, we find no significance in the treatment condition on students.

Thank you so much for any help, advice, comments!

$\endgroup$

1 Answer 1

0
$\begingroup$

You could compare effect sizes , in spite of samples being bit different, as long as you don’t have any suspicion that some of the estimates are biased due to some reason. For example, maybe in some samples you have some failed randomization with sample heavily skewed towards men or women.

However, if all subjects were participating in the same experiment, it would be much better to pool the observations and do regression with dummies for different groups. You will get more precision in larger sample.

$\endgroup$
4
  • $\begingroup$ Dear @1muflon1, thank you very much! Please allow me to ask a few new questions: 1) I combine all of the same data in one file and just one new column indication student, professional, or investor? 2) in the regressions do I use the qualifier if student ==1 (to distinguish between the three groups) 3) what about potential influences of retail investors when I analyze students -- by keeping all three separate (I thought) I avoided this. I am very grateful for your input. $\endgroup$ Feb 8 at 14:50
  • 1
    $\begingroup$ @hunter_gatherer 1) yes, and then include the dummy indicator in regression, you can also create the interaction term with whatever treatment you are testing. This is almost always preferable to running separate regressions because when you have two separate samples with lets say 250 observations you will have less degrees of freedom in each regression, which is very important for precision, compared to combining samples and adding extra dummies and interaction terms, each new regressor only costs you 1df so as long as the additional data has more observations than regressors you are $\endgroup$
    – 1muflon1
    Feb 8 at 14:57
  • 1
    $\begingroup$ always gaining precision and not loosing anything in terms of information (you just need to do tiny bit extra work calculating the effects from estimated beta + dummies which is trivial and takes only bit extra time). Doing this does not introduce any bias into the estimation per se. There are few situations where splitting the sample makes sense, for example, if you believe there is simultaneity in relationship only among investors but not students but those are rare edge cases $\endgroup$
    – 1muflon1
    Feb 8 at 14:59
  • 1
    $\begingroup$ 2) yes you include the dummies in regression, if you never did this before maybe google some basic info about regression with dummies/categorical data. Make sure you avoid the dummy variable trap, one group has to be baseline. 3) Any $\beta$ coefficient is estimated conditional on other regressors. As long as you properly include dummies and all relevant terms there won't be any bias. If you run $y = \beta_0 + \beta_1 T + \beta_2 I + \epsilon$ then $\beta_2$ will give you unbiased estimate of being investor on $y$ relative to student $\endgroup$
    – 1muflon1
    Feb 8 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.