Suppose we are interested in the effect of some 'treatment' on some outcome of interest. A common practice is to select a group of people and then randomly choose some fraction (often half) to receive the treatment. The remaining people are untreated (i.e. the 'control'). As shown in (e.g.) Angrist and Pischke, the difference in mean outcomes between the groups is then an unbiased estimate of the average causal effect of the treatment.
While such a procedure does give us an unbiased estimate, it does not ensure that the treatment and control group are 'balanced' on observable criteria. For example, although on average there will be the same fraction of men in the treated and untreated groups, in our actual sample there are likely to be a higher fraction of men in one group. In such a situation, it seems natural to simply take some men from the 'overly male' group and place them in the 'overly female group' until the gender ratio is the same in both groups. (If we wanted, we could select the men we choose randomly.)
- If we do this, will the difference in mean outcomes between treated and untreated groups remain an unbiased estimate of the causal effect?
- If so, why is this not done in all RCTs?
- If not, how do we trade-off the benefits of achieving 'balance' against the possible bias that we are introducing?
Optional extra: I am aware that this procedure seems rather similar to 'stratification'. However, my understanding is that, after stratifying, people do not compare mean outcomes for the treated and untreated groups (rather, they compare mean outcomes for every subsample on which they stratified, or equivalently run a regression with stratification dummies). If my proposal is indeed equivalent to stratification, my question is then equivalent to the question why we need these dummies and what are the costs and benefits of stratification.