1
$\begingroup$

Say each year I observe the effect of a movie-level treatment on the reviews of 10 movies (1 treated movie and 9 non-treated movies) over two periods (pre and post treatment). Each year, the treatment concerns 10 different movies.

How can I analyze those data using a diff-in-diff approach?

I suspect the conventional diff-in-diff approach from below to be wrong as it will pool together all the movies, without taking into account that they belong to different years/treatments:

$$ (1) y_{ijt} = \alpha_j + \phi_t + \beta I_{jt} + \epsilon_{ijt} $$ where $y_{ijt}$ is the outcome for review $i$ of movie $j$ on period $t$, $\alpha_j$ are movie fixed effects, $\phi_t$ is a dummy for the post-treatment period, and $I_{jt}$ is a dummy equal to 1 if movie $j$ is treated in the post-treatment period.

I also think the staggered diff-in-diff approach to not be suitable as (i) I have a repeated cross-sectional dataset and not a panel, and (ii) the treatment each time concerns different movies, meaning no movie is treated several times.

Thank you in advance for your kind help.

$\endgroup$
3
  • $\begingroup$ Please check if my understanding is correct. There are 10 movies. In "time" 1, movie 1 is treated and the others are untreated; in "time" 2, movie 2 is treated and the others are untreated, .... For every "time", data are collected over two periods (pre and post teatment) for the 10 movies. For example, if "time" means month, you have a data set collected over 10 months, and in month $k$, the $k$th movie is treated and the others are untreated. Every month, you have $y$ before and after the treatment for all 10 movies. Is that right? $\endgroup$
    – chan1142
    Jun 11, 2022 at 6:25
  • 1
    $\begingroup$ @chan1142 thank you for your answer. A small correction. In month 1, movie 1 is treated and movie 2 to 10 are untreated. In month 2, movie 11 is treated and movie 12-20 are untreated, and so on. So each month, there are different movies which might get treated or not. For each movie, I indeed observe $y$ before and after the treatment. So Intuitively speaking, I want to restrict the analysis to variations within each treatment (=month) so the outcome of movie 1 is only compared to movies 2 to 10, and not the outcome of the other movies. $\endgroup$
    – Nicolas L
    Jun 11, 2022 at 7:06
  • 1
    $\begingroup$ The idea would be for example to estimate Equation (1) for month == 1, month == 2,, monht == 3, and so on, and then take the average across those different regressions. I am looking for a way to achieve something similar with (1), as averaging accross regressions is not the best thing one can do. $\endgroup$
    – Nicolas L
    Jun 11, 2022 at 7:17

1 Answer 1

0
$\begingroup$

Based on OP's explanation in the question and comments, suppose that the movies treated and untreated are like

-----------------------------------------
month    treated movie   untreated movies
-------- --------------  ----------------
  1            1          2,  3, ..., 10
  2           11         12, 13, ..., 20
  3           21         22, 23, ..., 30
...
-----------------------------------------

and there are multiple reviews in each month. I guess OP wants a convenient way of getting the average treatment effect (averaged over multiple months). Suppose that data are pooled.

The method OP considers in his second comment to the question is (i) reg y i.tgroup##i.post x1 x2 if month==1, vce(r), etc. and then averaging the DID estimates. Let us see if we can do something similar using a single regression. Let's move step by step.

Consider the regression (ii) reg y i.month##(i.tgroup##i.post x1 x2), vce(r). Then (ii) should be the same as (i). Standard errors should be OK because heteroskedasticity-robust standard errors are used. The problem with (ii) is that there are $m$ (# of months) treatment effects, which OP wants to average.

So let us impose the restriction that the coefficient on the interaction term is common (the same over all months), and our regression is (iii) reg y i.month##(tgroup post x1 x2) i.tgroup#i.post, vce(r). Then a sort of "average treatment effect" is estimated, and all the rest coefficients are month-specific. So far so good.

Remarkably, (i) $\neq$ (iii) in general; (i) is an unweighted average of $\hat\beta_k$'s and (iii) is a particular weighted average of them. The reason follows. Due to the Frisch-Waugh decomposition, (i) equals $m^{-1} \sum_{k=1}^m \hat\beta_k$, where $\hat\beta_k = (X_k'M_{Z_k} X_k)^{-1} X_k' M_{Z_k} y_k$ (DID coefficient for month $k$), $X_k$ is the vector of the interaction terms for month $k$, $Z_k$ the matrix of all other regressors, and $y_k$ the vector of review rates in month $k$, while (iii) gives (according to algebra) $(\sum_{k=1}^m X_k'M_{Z_k} X_k)^{-1} \sum_{k=1}^m X_k'M_{Z_k} y_k = \sum_{k=1}^m C_k \hat\beta_k$, where $$C_k = \left( \sum_{j=1}^m X_j'M_{Z_j} X_j \right)^{-1} X_k'M_{Z_k} X_k.$$ Note that the mean-group estimator of (i) uses $1/m$ for $C_k$. (Note that $X_k$ is a column vector so $C_k$ is a scalar.) If $X_k'M_{Z_k} X_k$ is the same for all $k$ (months), then $C_k = 1/m$ and hence (iii) = (i).

$C_k\ne 1/m$ due to two reasons. First, the numbers of observations (reviews) are different across months. Second, the covariates (x1 and x2 in the regressions). (i) and (iii) are treatments averaged differently. I think it's a matter of choice. You could also consider other more restricted models such as reg y i.tgroup##t.post x1 x2, vce(r) as long as you can tell the difference. Why not?

I haven't rigorously checked the following.

If there are no covariates, we can change (iii) to WLS to construct the results from (i), where the weights are $1/\sqrt{n_k}$ with $n_k$ denoting the number of observations in month $k$. (Note: It's WLS to construct the mean-group regression. It has nothing to do with GLS.) That is,

by month, sort: gen nobs = 1/sqrt(_N)
reg y i.tgroup##t.post [aw = wgt], vce(r)

If there are x variables, then we can compute $C_k$ manually and construct the weights for WLS. Because $1/\sqrt{X_k'M_{Z_k} X_k}$ can be used as weights (for WLS) and $X_k'M_{Z_k} X_k$ is nothing but the SSR from the regression of $X_k$ on $Z_k$, the computation is conceptually straightforward, though Stata coding could be somewhat messy.

gen tr = tgroup*post
gen xmx = .
forv k=1/m {
  reg tr tgroup post x1 x2 if month==`k'
  replace xmx = e(rss) if month==`k' /* does it work? */
}
gen wgt = 1/sqrt(xmx)
reg y i.tgroup##i.post [aw = wgt], vce(r)
$\endgroup$
1
  • $\begingroup$ Thank you a lot, you answer was really helpful! $\endgroup$
    – Nicolas L
    Jun 12, 2022 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.