6
$\begingroup$

From Scott Cunningham in Mixtape, on propensity score matching:

propensity score matching has not seen as wide adoption among economists as in other non-experimental methods like regression discontinuity or difference-in-differences. The most common reason given for this is that economists are oftentimes skeptical that the conditional independence assumption can be achieved in any data set—almost as an article of faith. This is because for many applications, economists as a group are usually more concerned about selection on unobservables than they are selection on observables, and as such, they reach for matching methods less often.

Synthetic control seems simply to be a more sophisticated method of matching, where instead of matching each outcome with one observation per treated observation, we are matching based on a weighted average of observations.

Again, from Cunningham (on synthetic control):

What about unobserved factors? Comparative case studies are complicated by unmeasured factors affecting the outcome of interest as well as heterogeneity in the effect of observed and unobserved factors. Abadie, Diamond, and Hainmueller (2010) note that if the number of pre-intervention periods in the data is “large,” then matching on pre-intervention outcomes can allow us to control for the heterogeneous responses to multiple unobserved factors. The intuition here is that only units that are alike on unobservables and observables would follow a similar trajectory pre-treatment.

However, if I am weighting my observations by training on the pre-treatment period, is it not possible that there is some variable $X$ that will still confound and bias my estimates?

e.g. Say I want to evaluate the effectiveness of a medical intervention on lowering hospital costs. The likelihood of receiving treatment is based on a doctor's subjective assessment of how sick the patient will be in the future (unobserved confounding variable $X$). Now, I so happen to have 3 covariates on my patients: sex, age, and income. I use that data to select my optimal weights for the post-treatment period. So, in this case:

  1. Isn't synthetic control only applicable in this case insofar as my covariates are able to roughly approximate $X$? In this case, is it not entirely possible that they will not, even if I achieve a tight cross-validated fit in the pre-treatment period? -- i.e. the doctor knows something about the patients' future condition that my 3 variables cannot describe, even if my 3 variables are able to accurately reflect patients' costs pre-treatment.

  2. Additionally, how do I know that the relationship between the outcome of interest and my covariates do not change across time? Let's say sex, age, and income so happen to not be relevant for predicting patient hospital costs in 2022, even though they were quite effective in 2019 (potentially because of a confounding variable like a global pandemic). Wouldn't synthetic control completely misattribute the causal effect of the intervention? Or would this be moot using randomization inference?

$\endgroup$
1
  • $\begingroup$ I don't believe the above quote (or any other quote) says you are immune to unobserved confounders under synthetic controls. Could you narrow it down? $\endgroup$ Commented Jan 6, 2022 at 1:33

1 Answer 1

3
$\begingroup$

The answer appears to be in your quotes:

"The intuition here is that only units that are alike on unobservables and observables would follow a similar trajectory pre-treatment."

That intuition isn't necessarily true. There could always be unobserved confounders which hamper the analysis. Nothing renders you immune from that.

However, once you are done controlling for all observable factors, it seems unclear what else can be done. Most good econometric analyses contain a large narrative component indicating to readers reasons why a given DiD (or other treatment method) is plausibly randomized or the nonrandom component is sufficiently accounted for. This element is not statistical in nature - it is a persuasive narrative designed to convince (or illustrate to the reader) that all unobserved factors are controlled for, something that cannot be done by pointing to observed data.

If all observable factors are controlled for, and some respectable attempt at identifying uncontrolled factors has been shown, then I would expect any remaining naysayers should at least be able to name specifically what unobserved factor is causing the problem. Otherwise, a critic could always suppose some omitted variable exists, and then add no further commentary.

$\endgroup$
3
  • 2
    $\begingroup$ via Wikipedia: "The Achilles' heel of DID is when something other than the treatment changes in one group but not the other at the same time as the treatment, implying a violation of the parallel trend assumption." While the stated benefit of synthetic control is that it does not require the parallel trends assumption, this admission seems to imply that synthetic control is no better than DID in situations where there might be "treatment changes in one group but not the other at the same time as the treatment." $\endgroup$
    – Daycent
    Commented Jan 6, 2022 at 23:50
  • $\begingroup$ your answer makes sense, though it is not entirely satisfactory -- it seems the onus should be on the researcher to ensure that there are no unobserved factors that affect the results. $\endgroup$
    – Daycent
    Commented Jan 6, 2022 at 23:51
  • $\begingroup$ That's unfortunately not possible. The researcher can address many unobserved factors, but not all conceivable ones. One cannot disprove the existence of a thing, particularly an unobserved thing. $\endgroup$ Commented Jan 7, 2022 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.