Let there be a policy $T$ which can either be implemented or not implemented by an organisation. Suppose there are $n$ organisations and the outcome variable $Y$.

The policy is not centrally imposed- different organisations have adopted the policy on their own will at different points in time. Which suggests that there is a significant possibility of self-selection bias.

I wanted to know if there is any technique to find out if such a policy is effective or not.

I have read about Difference-in-Difference and Discontinuous Regression-- but as far as I have understood-- they need a policy implemented at a particular point in time, and two groups, one of which is treated and the other is not.

In my case, there are two groups-- one that is treated and one that is not treated, but the time of implementation is different for different organisations.

I may be unclear, so I present a simplistic example. Suppose there are 6 firms A, B, C, D, E and F, having all walls in their respective offices white or green. The $Y$ variable of interest to me is the total productivity of employees (suppose I have a way to measure that perfectly) for example.

In the beginning (suppose in the year 2000) all the firms had white walls. Then in 2003, firms C and E changed to green. Then in 2005 firm B also changed to green. The other firms have not made any changes till now (say, 2010).

I have the employee productivity data (monthly) of all the five firms from 2000 to 2010 and also the data of when they switched from white to green.

Is their any method to find out if green walls have any advantage over white walls with this data? I am aware of the possible self selection bias (maybe the firms decide to change to green when they see an unusual drop in productivity etc.) Is there any way to overcome this and evaluate if the policy of switching to green works?

If yes, where can I read more about it, the assumptions, how to test them, then how to actually conduct the analysis using software etc.


The description is bit vague on how many organizations we are talking but you could still do differences-in-differences (DiD) for all treatment-control-pairs that adopted it at the same time separately and then running meta-analysis on the results.

As far as I understand based on Angrist & Pischke (2009) Mostly Harmless Econometrics ch.5, DiD solves any possible self-selection bias as long as the identifying assumption of common trend is satisfied. That is in DiD it it is required for both treatment and control to follow common trend pre-treatment. If the changes in outcome variable ($y$) pre-treatment are virtually same between treatment and control, then even if the levels of $y$ are different the estimates won't be biased. This of course will have to be tested in each case.

The above will however only be feasible if you do not have to run too many of the individual DiD otherwise it would be unpractical.

An alternative could be the event study design (see MacKinlay, 1997) that is used quite often across finance literature. It allows for events to occur at different times and it does not matter that firms/organizations experiencing the event are heterogenous and possibly self-select for the event (this approach is often used to evaluate M&A effect on stock returns for example), but on the other side this approach is very data intensive and depending what precisely is $y$ it might not be appropriate, or also if there is event clustering.

In order to implement regression discontinuity design (RDD) there must be some cut-off point that determines whether treatment is assigned to subject or not. For example, RDD would be appropriate if there is a policy that requires any power plant producing more than let's say 1.21 gigawatts is required to adopt some $CO_2$ curbing technology and below that threshold it is not required, and you want to examine the effect of such policy on some outcome variable like air quality. There is also so called fuzzy RDD which does not require sharp cut-off in some deterministic variable as long as probability of adopting the treatment is significantly different (see ibid. (2009) Mostly Harmless Econometrics ch.6). The feasibility of this will again depend on data availability and other details, but if the organization themselves have full control over probability of treatment it will not be appropriate.

Lastly, as you could have probably already guessed from the discussion above the answer the choice of methodology is always context dependent. In order for anyone to give you concrete answer you would have to share virtually every single detail of the study which is not feasible, so take the above as some pointers with references (since you tagged the question as a reference request) rather than telling you exactly what to do.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.