6
$\begingroup$

I'm using the synthetic control method to evaluate a state-level policy (using synth package in Stata). I've read the seminal ADH papers on CA tobacco & Germany reunification, but I still don't know exactly how to decide if the constructed synthetic state is a good fit. What we do at the moment is looking at the graph - to see if there's overlap between the synthetic and the real state. However, I feel this is not a very "robust" way...

Is there any measure that we can use or test on?

Edit:

After reading through reply (and papers mentioned again), I have another related (but maybe a bit nuanced) question on what to expect at the time of (or immediate after) intervention.

As shown in the two following figures from ADH (2010) and ADH (2015), there is a small effect in the same year of the intervention for CA tobacco Propostion 99, while in the Germany reunification study, the gap only becomes noticeable 2 years later. In some other papers using SC, there is an immediate effect (such as Cavallo et al. (2013)). I suppose as long as there is a consistent story to support the results, the short-term effect size of the intervention will not invalidate the fitness of synthetic control?

ADH (2010) CA tobacco ADH (2015) Germany reunification

$\endgroup$
  • 1
    $\begingroup$ I edited my question in response to your edit, but if you have multiple questions its better to ask them separately rather than keep extending original question. Especially, when the second question, although related, is distinct from the original one. Questions here should not become too broad since such questions are not good fit for the site. See the sites guidance on asking question: economics.stackexchange.com/help/on-topic $\endgroup$ – 1muflon1 Jun 8 '20 at 22:31
3
$\begingroup$

This is an excellent question, but unfortunately with no (yet) agreed upon answer.

Many authors such as in those seminal papers use cross-validation for this. This is basically what is implied in the ADH (2015) paper on German unification. If the synthetic control is good the MAPE (mean absolute prediction error) and RMSPE (root mean square prediction error) should be quite low. Additional cross-validation where the pre-treatment period is split into training and testing period is so far the common way to go, although there is no agreed upon value of how low the prediction errors should be, just that lower is always better.

Next it is also argued that examining if the trend in the countries from donor pool that are assigned non-zero weights and seeing if the pre-treatment trend in those donor countries seem to match the pre-treatment trend of treated country similar way as you would do for DiD (Difference in Differences) as argued here.

Lastly, as crazy as it may seem, yes visual inspection is actually recommended as well.

Beside the above you should also conduct sensitivity analysis by systematically trying to vary the number of countries in the donor pool and see if the resulting synthetic control has still a good fit. That's again also done by ADH 2015 and you will find it in almost all studies as well. Publishing your weights is also considered good practice.

Of course, doing a proper cross-validation is difficult if you have small pre-treatment sample. I was once working on a synthetic control study at one of the euro-system member's central bank with few colleagues where we had very short pre-treatment period due to data constraints. Instead of doing cross-validation we decided to just test whether the real pre-treatment series is statistically significantly different from the synthetic series using couple of non-parametric tests. When we presented the results and methodology at a workshop no objections to that were raised, and there were some very excellent econometricians present in the room. However, beside central bank's brief the study was not published because the results were not very interesting as placebo test showed results were not significant and we decided to abandon the project since we judged chance of publication to be low so the idea never got proper peer review, so take that as you will.


Response to edit:

as with classical DiD there is always an issue of potentially delayed effects or anticipation effects that would bias the result. However, in this case the authors argue that the increase after the reunification was actually part of the effect and not delayed effect. As the authors say:

Our estimate of the effect of the German reunification on per capita GDP in West Germany is given by the difference between the actual West Germany and its synthetic version, visualized in Figure 3. We estimate that the German reunification did not have much of an effect on West German per capita GDP in the first two years immediately following reunification. In this initial period, per capita GDP in the synthetic West Germany is even slightly lower than in the actual West Germany, which is broadly in line with arguments about an initial demand boom (see, e.g., Meinbardt et al. 1995). From 1992 onward, however, the two lines diverge substantially. While per capita GDP growth decelerates in West Germany, for the synthetic West Germany per capita GDP keeps ascending at a pace similar to that of the prereunification period.

However, arguably this is a weak point of the paper.

Otherwise, if there would be a genuine delay in the effect you would have a problem that could be solved similarly as with DiD by identifying the 'correct' date where the treatment started to have an effect and identify the pre and post treatment period based on that.

$\endgroup$
  • $\begingroup$ Thanks for the quick reply! also, can you provide more details on the non-parametric tests you mentioned? $\endgroup$ – deeyuan Jun 8 '20 at 20:11
  • $\begingroup$ @deeyuan we did wilcoxon rank sum test permutation test and bootstrap - we did only these because our pre-treatment samples were only about 20-24 observations depending on variable, I don’t want to say that doing non-parametric test here is the way to go necessary if you have some nice sample size there some better ways how to compare two time series for sure $\endgroup$ – 1muflon1 Jun 8 '20 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.