1
$\begingroup$

Suppose in a diff-in-diff problem, I have two control groups, but just one can be used as comparison, so I want to check which of them provides a better comparison. My idea is to compare the parallel trends in the pre-treatment period. The one who with the closer trend (even if none of them are strictly parallel) will be chosen.

E.g:

enter image description here

How could I, mathematically, compare groups 1 and 2 to check which one provides the "more parallel" trend to the treatment group?


EDIT:

Jus to be more specific, I don't want to TEST for the parallel trends. I want to compare two trends, and if if none of them are good enough, I want to decide which one is "more parallel". The motivation for this is this paper. It's explained in details in this post. They are estimating diff-in-diff but the control group is composed by non-treated units weighted by similarity to the treated units.

To check the results they suggest comparing one series (classical control group) with another series (their approach) which is the same group weights, so estimated via WLS. Weighting can improve the results, but there are specific situations in which just homogenous weights can be better. They metion this, but do not provide a way to compare.

$\endgroup$
3
  • $\begingroup$ Can you point me to where the authors decide which trend is more parallel? I've skimmed through the paper and it appears that the authors improve the method I've mentioned to you. It appears Athey and Imbens were excited for the synthetic control method (SCM) back in 2017 and together with their colleagues decided to further improve this estimator for causal effects with panel data. I am still not entirely sure what is your question here. $\endgroup$
    – bajun65537
    Feb 3, 2023 at 10:54
  • $\begingroup$ On a separate note, I think the other resource you've shared 'Causal Inference for The Brave and True' is one of the best I've seen. Python code supporting every chapter is extremely useful. $\endgroup$
    – bajun65537
    Feb 3, 2023 at 12:11
  • $\begingroup$ @bajun65537 you are right, they came up with an extensions for the SCM. But this extension allows for fit the synthetic control in trends instead of level. In the end they fit a new series and do DID. Basically they do Diff-in-Diff with weighted least squares. So we need to now if this weighted series is really superior to a series with homogenous weights. They do not mention n the paper how to compare, but mention that users shoud take a look on both to decide $\endgroup$
    – Oalvinegro
    Feb 8, 2023 at 0:57

2 Answers 2

1
$\begingroup$

I'm not understanding-- could you please talk a little about why both can't be used here? It's been a while since I've really worked with DD, but as far as I'm aware, there's not real metric you can point to. Unlike say in synthetic controls, where the $RMSE$ is typically a candidate for the quality of the pre-intervention counterfactual, DD has no such metric (again, as far as I'm able to remember). Why not use both comparison groups, though?

Strictly speaking, if you wanted to go wild, there are computer science/engineering algorithms that do this sort of thing (dynamic time warping, others), but this is pretty much never done in DD/econ and would be a little like taking a nuclear bomb to a high school bully. If I were you, I would make strong conceptual arguments for the merits of both control groups and compare the statistical results you get and what possible impact the different groups might have.

$\endgroup$
2
  • $\begingroup$ I built on your answer, @Jared Greathouse. I liked that you mentioned the synthetic control method! $\endgroup$
    – bajun65537
    Jan 27, 2023 at 20:30
  • $\begingroup$ Thnak yyou for your comment, I edited my question to clarify the situation and why I cannot use both groups. $\endgroup$
    – Oalvinegro
    Jan 31, 2023 at 1:04
1
$\begingroup$

Background

The typical thing to do is visually inspect the pre-treatment trends for the control and treatment groups. Whether you want it or not, you might be biased when looking at the visual representation. I'd argue that an average university student would claim that neither control 1 nor 2 pass the "parallel trend" assumption. At the same time, many researchers do a more relaxed visual inspection, given that you explain the theory and arguments that support the pre-treatment parallel trends. This, of course, causes some friction, e.g. look at Kearney and Levine (2015) and Jaeger, Joyce and Kaestner (2018), and then again Kearney and Levine (2018).

Mathematical methodology

It is possible to test for the parallel trend assumption more concisely. This answer provides a very nice intro on how to do it.

A formal test which is also suitable for multivalued treatments or several groups is to interact the treatment variable with time dummies. Suppose you have 3 pre-treatment periods and 3 post-treatment periods, you would then regress $$y_{it} = \lambda_i + \delta_t + \beta_{-2}D_{it} + \beta_{-1}D_{it} + \beta_1 D_{it} + \beta_2 D_{it} + \beta_3 D_{it} + \epsilon_{it}$$ where $y$ is the outcome for individual $i$ at time $t$, $λ$ and $δ$ are individual and time-fixed effects (this is a generalized way of writing down the diff-in-diff model which also allows for multiple treatments or treatments at different times).

The idea is the following. You include the interactions of the time dummies and the treatment indicator for the first two pre-treatment periods and you leave out the one interaction for the last pre-treatment period due to the dummy variable trap. Also now all the other interactions are expressed relative to the omitted period which serves as the baseline. If the outcome trends between treatment and control group are the same, then $β_{−2}$ and $β_{-1}$ should be insignificant, i.e. the difference in differences is not significantly different between the two groups in the pre-treatment period.

What to do?

Always present a graph showing the levels of the two series you are comparing over time, not just their difference. The general rule of thumb is to prefer DiD on a matched sample for this reason – if you can make the levels more similar, readers will be more willing to think the trends will be too.

Kahn-Lane and Lang (2019) on the "failure to reject parallel trends in the pre-treatment data".

Increasingly, researchers point to a statistically insignificant pre-trend test to argue that they therefore accept the null hypothesis of parallel trends. There is no doubt that testing for a common pre-trend plays an important role in validating the parallel trends assumption underlying DiD. However, failing to reject that outcomes in years prior to treatment exhibit parallel trends, should not be confused with establishing the validity of the parallel trends counterfactual. Moreover, clearly, not rejecting the null hypothesis is not equivalent to confirming it.

But also, and in my opinion, more importantly, Kahn-Lane and Lang (2019) note that:

Authors should perform a thorough comparison of the differences between the treatment and control groups including demographic composition, other factors that could have differentially affected each group, and comparison of trends as far back as possible .

Relax parallel trend with Synthetic Control Method

Assuming that the number of pre-intervention periods is large enough, it is likely that unobserved and time varying confounders have similar effects on both treated unit and synthetic counterpart (Kreif et al., 2016). In your case, the synthetic control method might improve the DiD approach by relaxing the parallel trends assumption. However, the longer the pre-treatment data, the better. Moreover, with the DiD there often exists a sort of ambiguity when it comes to the choice of control group as argued by Card (1989) - in case of SCM it is being chosen by a data-driven method.

If you want to read more on Synthetic Control Method, I highly recommend going to the source, i.e. Abadie et al. (2010), Abadie et al. (2011) and Abadie et al. (2015). The application of the SCM for estimating treatment effects in panel settings has become very popular among researchers interested in comparative case studies. Since then, the SCM was applied in various fields and to diverse research topics, establishing an intuitive alternative to creating counterfactuals. According to Athey and Imbens (2017):

Synthetic Control Method is arguably the most important innovation in the policy evaluation literature in the last 15 years.

Therefore, using SCM as the robustness check for DiD is more than justified.

$\endgroup$
1
  • $\begingroup$ Thank you for your comment, I edited my question to clarify that I'm not looking for test the parallel trends $\endgroup$
    – Oalvinegro
    Jan 31, 2023 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.