Why we need at least 40 groups to be properly clustered?

Question

From this discussion, I deem that we need approximately 40 groups for clustering.

For example, if we want to clustered by industry, we need at least 40 industries, or if we want to cluster by year, we need at least 40 years. I am wondering if there is any reference or justification for this?

But in this paper, Table 11, the author clustered a multi-way cluster (industry and year). However, their sample is from 1990 to 2012, less than 40 years, so how come they can do cluster by year in this case based on the above justification?

Answer 1

First of all, clustering only changes the (estimated) standard deviations of your coefficients. Whether you cluster or not will not change the coefficients themselves and therefore also not the "true quality" of these estimates.

If you cluster, this means that you assume the observations within a given cluster are not independent. If this is true, then the true number of independent observations only equals the number of clusters. So if you have $N$ observations and $G$ clusters and if your data is not independent within each cluster, it is like you only have $G$ observations to work with.

Given that, for example, estimated standard deviations of your estimator are based on large sample statistics, this means that you need enough clusters (independent observations) in order to obtain a "good enough" approximations for your standard deviations.

There is this "old" rule of thumb in statistics that large sample statistics start to "work" as soon as you have 30-40 independent observations. This however is an arbitrary number and there is not reason why you should stick to it. The general rule is: the more independent observations, the better.

If you are afraid that your data is not independent within your clusters, and if the number of clusters is not big enough, the best option would be to get more data. If that's not possible, I guess I would run the regression with and without clustered st-dev and report both.

Answer 2

First, that is only a rule of thumb, exact number of clusters you need in any case is context dependent.

Second, actually as pointed out by Cameron et al (2008):

A practical limitation of inference with cluster-robust standard errors is that the asymptotic justification assumes that the number of clusters goes to infinity.

Consequently, clustered standard errors only provide correct inference asymptotically. However, if you ever took any statistics class you might already know that the sufficiently large samples for asymptotic results to hold are typically samples with sample size of 30-40 observations. For example, for standard regression a typically a sufficient sample size to justify its asymptotic properties is 30 observations per independent regressors (see Verbeek A Guide to Modern Econometrics pp 36).

For a clustered errors a rule of thumb is 30-40 (clusters i.e. groups don't confuse this with observation count) although most authors go with the upper value of 40 or even Angrist and Pischke went for 42 (see MHE pp 319*). Now the rule is arbitrary in the sense that of course every problem is unique and sometimes 29 or 31 clusters might be sufficient and sometimes you might need even 45 clusters (the other answer is correct in pointing out more is always better).

However, the number is not arbitrary in a sense its completely made up. The number of 30-40 cluster is actually based on examination of how different errors affect statistical inferences from Monte-Carlo simulations such as the one below (table taken from Cameron et al 2008).

You can see that as number of clusters increases the rejection rates (and their standard deviation) drops. Already with 30 clusters you can see that clustering errors are doing pretty well but depending on a setting more clusters might be needed still.

Minimum of 40 clusters is very reasonable rule of thumb for an applied econometrician. If you would want to get to exact number you would have to do some Monte-Carlo simulations for your problem and most likely you would find that the result is very close to the rule of thumb. Consequently, if you are willing to tolerate some (very) small possibility of having incorrect inference minimum of 40 clusters is reasonable rule of thumb (again more is of course even better). If you are willing to tolerate larger but still small possibility of having incorrect inference you can even use 30 as a rule of thumb.

_{* Althought that was partially for the lolz since they constantly reference Douglas Adams The Hitchhiker's Guide to the Galaxy}