Self-selection bias during the course of experiments

Question

Suppose you are running a randomized experiment to assess the effect of $X$, say some training program for unemployed people, on $Y$, say the chance of finding a job in the coming year. Suppose also that $X$ takes time : maybe it lasts for several month.

Because you randomize, you do not need to worry about self-selection bias initially. But during the course of $X$, some people will likely realize that $X$ is beneficial to them, and others may realize that they are wasting their time.

As a result, one might expect that among people who drop from the program, there is a higher proportion of agents for which the treatment effect would have been smaller. This might induce an over-estimation of the treatment effect.

My questions are :

• Is this kind of bias discussed in the literature on randomized experiments?
• Does it have a canonical name ?
• Do researcher try to control for this, and if yes, how?

Answer 1

Apparently this is called attrition bias. It's very similar to survivorship bias. This paper suggest correcting for it using Heckman correction. Propensity score matching may also help somewhat. My experience with both has been mixed, but they are commonly used. You should figure out what exact approach is most appropriate for your setting.

One last edit: These two papers, which talk about bounding the average treatment effect, may also be of use to you.

Answer 2

I think this paper might be useful to you. It's a job market paper by one of Heckman's students at UChicago, named Rodrigo Pinto. The paper is titled "Selection Bias in a Controlled Experiment: The Case of Moving to Opportunity." In the MTO experiment, the voucher assignment mechanism was random but only approximately half that received the voucher ended up actually moving. This creates a problem because the usual analysis (treatment-effect-on-the-treated or intent-to-treat) will only tell us the causal effect of receiving a voucher. However, we are interested in the causal effect of the new neighborhood, not of receiving the voucher. He shows how to decompose the typical treatment-on-the-treated parameter into components that have unambiguous interpretations. Namely, he isolates the causal effect of the new neighborhood.

Answer 3

Another thing you can look at is "Intention-to-treat analysis". From Wikipedia,

An intention-to-treat (ITT) analysis of the results of an experiment is based on the initial treatment assignment and not on the treatment eventually received. ITT analysis is intended to avoid various misleading artifacts that can arise in intervention research such as non-random attrition of participants from the study or crossover.

This appears to match what you were looking for: your treatment is randomized initially and people drop out non-randomly.