Randomisation and balance in experiments

Question

Suppose we are interested in the effect of some 'treatment' on some outcome of interest. A common practice is to select a group of people and then randomly choose some fraction (often half) to receive the treatment. The remaining people are untreated (i.e. the 'control'). As shown in (e.g.) Angrist and Pischke, the difference in mean outcomes between the groups is then an unbiased estimate of the average causal effect of the treatment.

While such a procedure does give us an unbiased estimate, it does not ensure that the treatment and control group are 'balanced' on observable criteria. For example, although on average there will be the same fraction of men in the treated and untreated groups, in our actual sample there are likely to be a higher fraction of men in one group. In such a situation, it seems natural to simply take some men from the 'overly male' group and place them in the 'overly female group' until the gender ratio is the same in both groups. (If we wanted, we could select the men we choose randomly.)

Some questions:

• If we do this, will the difference in mean outcomes between treated and untreated groups remain an unbiased estimate of the causal effect?
• If so, why is this not done in all RCTs?
• If not, how do we trade-off the benefits of achieving 'balance' against the possible bias that we are introducing?

Optional extra: I am aware that this procedure seems rather similar to 'stratification'. However, my understanding is that, after stratifying, people do not compare mean outcomes for the treated and untreated groups (rather, they compare mean outcomes for every subsample on which they stratified, or equivalently run a regression with stratification dummies). If my proposal is indeed equivalent to stratification, my question is then equivalent to the question why we need these dummies and what are the costs and benefits of stratification.

Answer 1

Ok, I am far from an econometrician, but my train of thought would be as follows:

By using random assignment we have two groups that are on average equal to one another on all aspects. As you rightly point out due to sampling variation there will be differences.

My worry would be that if I start "correcting" those after randomization (as per your example with gender distribution) I introduce other differences between the groups in say age or even an unobserved but important characteristic.

Stratification is different as it is done upfront. You have a characteristic that you know is different for different groups so you take the random samples within those groups. As a rather naive example: we know that the effect of taking anticonception pills is rather different for men and women, so if you wanted to test the effect of the pill on fertility you could stratify by sex and then do a random assignment within the sexes.

Answer 2

The idea is that for causal inference you want to be able to treat the control group as a counterfactual. In the ideal world you would wanna be able to observe parallel world where the same people who you give treatment don’t get one.

Now simple random sample promises to deliver this because as long as everyone has really random and most importantly equal chance to be included then the distribution of unobservables should be the same between groups and you should have no bias.

However, if you are taking random samples (especially if they are not really big) you may end up by a chance with a sample where let’s say 90% of control is only men and only 10% women. In that case you may wanna balance this out because then the control may not really be good counterfactual.

This does not mean that you can balance it in whatever way you want. People should be moved between groups on random to avoid bias or even better both control and treatment groups should be mixed and you should randomize again and then check the distribution. If the number of men/women between treatment/control is not statistically significantly different proceed with the experiment.

The goal is always to create the best counterfactual possible. This could be actually done even without simple random sampling at all for example by using propensity score matching.

Sometimes you may even need to conduct/observe field experiment where randomization is not possible, but there are still techniques that attempt to find a good counterfactual (you may wanna look at differences in differences or synthetic control approach).

In fact in simple OLS biased coefficient can be expressed as:

$$ E[\beta]= \beta + \gamma \frac{cov(x,e)}{var(x)}$$

Where the second term is bias. Hence to get rid of bias you want to make sure the independent variable is also independent on error term $cov(x,e)=0$. Simple random sampling can help you achieve that but it’s not panacea. If sample is highly unbalanced with just by chance all women being in control and men in treatment, then treatment won’t be really independent of unobservables.

Answer 3

A small rant about balance tests:

You have two groups, one treated and one not. You randomized assignment into both groups. Now conduct a t-test for differences in means between these two groups. What are you testing? Yes, how likely the differences between these groups are due to pure chance if they came from the same distribution. Do we need to test this? No, we know that is true because we randomly pulled participants from our sampling frame to start with.

It remains a mystery to me why Angrist and Pischke included this lengthy discussion about balance tests in the MHE section on the experimental ideal. If anything, issues such as non-random take-up or attrition should be reasons to worry, but not the differences between the two assignment groups.