0
$\begingroup$

I am trying to implement a difference in difference model. Suppose that some treated units anticipate the treatment. Then, that would mean that they respond to the treatment prior to the treatment actually occuring. Since the treatment effect would then be present in the "pre" period, it seems like this would almost always bias the relative post-period treatment effect downwards in magnitude.

In this sense, it never really matters if there is an anticipation effect. If there is an anticipation effect, all it does is decrease my Type 1 error rate and increase my Type 2 error rate. Given publication bias often results in a high Type 1 error rate (I discover some treatment effect that doesn't exist), isn't the "anticipation effect" really a moot point?

Thanks in advance!

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ There's no reason to assume the two biases are of remotely equal sizes, or even estimable sizes in the population at large. Nor will there (usually) be evidence that in any particular case one bias isn't dominating. When considering opposing biases, it's prudent to assume you're wrong in both directions. $\endgroup$
    – RegressForward
    May 6, 2022 at 15:02

3 Answers 3

Reset to default
1
$\begingroup$

[a synthesis of Evangelos, Michael and my comments / posts]

You are right if you have a simple, two-time period diff-in-diff and you believe parallel trends is true OR if you have staggered/heterogenous timing with comparator group only being the never treated.

But, if we sometimes compare the treated with different groups of pre-treated (as in staggered/heterogenous timing with comparator group including the pre-treated), then we could get a negative coefficient with anticipation effects.

As a motivating example, say we're evaluating effects of cash transfers, and we only have two time periods. Some (very poor individuals) anticipate they will receive a cash transfer, and therefore spend before receiving the cash transfer to help improve their livelihoods. Others do not anticipate and spend to a lesser degree after receiving the cash transfer. Then, since treatment timing is homogeneous (rich and poor treated at the same time), the diff in diff is:

[Average of treated post] - [Average of treated pre] - ([Average of control post] - [Average of control pre])

The treated comprise both rich and poor. Anticipatory behavior generally means that [Average of treated pre] is larger.

However, you'd get a negative effect if somehow there was heterogeneity in treatment timing. Suppose instead that the rich were treated first, while the poor were untreated. The poor anticipated treatement, and increased spending while untreated. Then the comparison between treated and pre-treated will result in the pre-treated having increased spending a lot (due to anticipation) and a potentially negative estimate. This gets resolved if you restrict the comparator group to the never-treated.

I'd still be wary, however. It is possible that anticipatory behavior means that [Average of treated pre] is smaller (take moral hazard with health insurance, for example).

Moreover, treatment effects can incorporate (1) a change in levels and (2) a change in trends. If treatment (and anticipation of treatment), causes a change in trends, then parallel trends will fail, causing problems. In general, pre-trends due to anticipatory behavior can't be distinguished from pre-trends due to any other reason, which would imply a failure of parallel trends.

Also, nice review article of the literature arxiv.org/abs/2201.01194.

Improve this answer
$\endgroup$
1
-1
$\begingroup$

If some units anticipate the policy change and act before others, then it seems to me that this problem is almost isomorphic to staggered adoption DiD design. And that literature suggests that DiD with heterogeneous effects can get the estimates very wrong (e.g. wrong sign). So, the type of anticipation you are describing would be a serious concern---for example, those anticipating are probably the ones that have the most to gain.

So, I doubt that it will nicely offset publication bias.

Improve this answer
$\endgroup$
14
  • $\begingroup$ The paper being referenced is this one, sciencedirect.com/science/article/pii/S030440762030378X $\endgroup$
    – Michael Gmeiner
    May 6, 2022 at 7:47
  • $\begingroup$ If there is heterogeneous treatment timing and heterogeneous treatment effects, then the two-way fixed effects estimator has bias. $\endgroup$
    – Michael Gmeiner
    May 6, 2022 at 7:48
  • 1
    $\begingroup$ The original question was about some (and not all) treated unites anticipate the treatment and respond. That's what makes me think it's similar to staggered treatment. In the simplest case, I think would agree with your other answer. But heterogeneous effects are (rightly) most dominating the literature. $\endgroup$
    – Evangelos Con
    May 6, 2022 at 12:16
  • 1
    $\begingroup$ In Daycent's example, is there a control group without cash transfers? Are rich and poor treated at the same time? If so, the diff in diff is: [Average of treated post] - [Average of treated pre] - ([Average of control post] - [Average of control pre]). The treated comprise both rich and poor. Anticipatory behavior means that [Average of treated pre] is larger. However you'd only get a negative effect if somehow there was heterogeneity in treatment timing. $\endgroup$
    – Michael Gmeiner
    May 6, 2022 at 14:04
  • 1
    $\begingroup$ That is, suppose the rich were treated first, while the poor were untreated. The poor anticipated treatement, and increased spending while untreated. Then the comparison between treated and untreated will result in the untreated having increased spending a lot (due to anticipation) and a potentially negative estimate. But if you have homogeneous treatment timing, I'm not convinced this problem persists. $\endgroup$
    – Michael Gmeiner
    May 6, 2022 at 14:25
-1
$\begingroup$

If you have a simple, two-time period diff-in-diff and you believe parallel trends is true, I think you are right.

I'd still be wary, however. Treatment effects can incorporate (1) a change in levels and (2) a change in trends. If treatment (and anticipation of treatment), causes a change in trends, then parallel trends will fail, causing problems.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I'm not convinced Evangelos's answer is correct. In your specification, all the units have the same treatment time correct? There's no heterogeneity in treatment time, just some have anticipation? The paper that Evangelos cited has the core issue of heterogeneity in treatment time. If you don't have that, then I'm not sure that answer is correct. $\endgroup$
    – Michael Gmeiner
    May 6, 2022 at 13:57
  • $\begingroup$ I originally was considering the general case (both heterogeneity and homogeneity in treatment timing). my apologies for not clarifying in the question. $\endgroup$
    – Daycent
    May 6, 2022 at 14:44
  • 1
    $\begingroup$ @Daycent Seems that Michael and I focused on different parts of your question. Given that you also state that you wanted an answer for both heterogeneity and homogeneity in treatment timing, I am not sure that this should be the accepted answer. I think your synthesis is a more appropriate answer for the original question. And perhaps consider adding a line or two about heterogeneous effects. Also, perhaps address your Q about pub bias (see my answer and RegressForward's comment), and Michael's comments below are more informative that this answer in my opinion. $\endgroup$
    – Evangelos Con
    May 7, 2022 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.