Difference-in-differences using countries: is it necessary to cluster by country?

Question

I have a question regarding difference-in-differences (DD) studies. If I want to compare a policy that is implemented differently in three different countries - thus three different treatment groups - using a DD approach, is it necessary to cluster the standard errors by country (instead of, for example, by industry)?

From my understanding, this would make a DD approach almost impossible, as the number of clusters is not high enough (e.g., not higher than 42 or whatever rule of thumb evaluators use in practice). In other words, all DD studies comparing policy effects between 2 or 3 different countries (or states) would be invalid.

Answer 1

You should cluster on level where you believe the autocorrelation or heteroskedasticity occurs.

If you don’t have enough clusters at a given level, for example, if you believe the above mentioned problems occur on country level but you have only 3 countries, you are correct you should not be using clustered standard errors which by rule of thumb require at least about 40 clusters to be properly estimated. However, clustered errors are not the only type of error adjustment there is. For example, you can instead of clustered errors use Newey-West errors or bootstrapped errors (of course these have their own advantages and disadvantages as well). The above mentioned are just examples other ways of adjusting errors exist as well.

Answer 2

In general, you should look to cluster-adjust standard errors at the level in which you believe there exists autocorrelation/heteroscedasticity of the errors if you don't wish to (or can't) explicitly model the structural correlations within the regression variance-covariance matrix.

In the context of a panel DiD, you may also be compelled to model fixed effects at the ID level since fixed effects would account for only part of the unobserved heterogeneity, namely, the portion of it that is time-invariant. Doing so lessens the statistical burden needed to reach consistency of the estimators (relative to clustering standard errors) and accounts for this heterogeneity explicitly in your specification in exchange for reducing the amount of variation available to identify model parameters. It's generally a good idea, however, to consider modeling fixed effects in a panel DiD if you believe this sort of heterogeneity is present, unless your theoretical framework gives you plausibly exogenous covariates that purport to address omitted variable bias that is typically easy to argue exists.

I find it helpful to think of two common scenarios in econometric work where autocorrelation of the unobserved heterogeneity within clusters occurs, in which case you should then consider clustering the standard errors at that level:

1). Firstly, consider the sampling design of an experiment consisting of randomly picking a subset of clusters of individuals from a population of clusters you wish to inference for. For example, let's say you randomly sample individuals from 10 counties that you randomly choose from within a state. In order to make inferences for the state at large using the coefficients estimated from a regression using only the 10 counties at your disposal, you should cluster the resulting standard errors. Clustered standard errors here are roughly equivalent to their counterpart heteroscedastic-consistent Huber-White standard errors if there is no heterogeneity in the treatment effect.

2). If clusters of individuals are assigned the treatment, as opposed to the individuals themselves, then you should cluster standard errors at that level. Treatment at the individual level does not require clustering unless you have repeated instances of individuals over time, in which case you should then cluster at the individual level.