I apologize in advance for a long and possibly misworded question. I am trying to get to something but I am not exactly sure how to explain it. Hopefully, you can bear with me and provide me with some guidance.

I am interested in the trade-off between observational studies with a large set of controls and quasi-experimental studies with fewer controls.

I often see quasi-experimental papers that (try to argue they) get some almost exogenous variation, but at the expense of having fewer data points and fewer control variables. For example, you often see people rely on a new state-law to induce changes in a treatment-variable. To analyze the treatment, they then use a yearly county-level panel-data over, say, a 10 years period including the adoption of the law. Looking within this single state over that a period may allow them to argue that the variation is almost exogenous. But it leaves them with few county-controls to mitigate for possible endogeneities (e.g., in the case of US counties, because yearly county data is only available from the ACS for counties of more than 65,000 people). That is, their claim to exogeneity relies mainly on the quasi-experimental nature of the treatment, and little on controls (conditional exogeneity).

Another option would have been to run a purely observational study with a larger set of controls and put more of the weight of causal inference on getting closer to conditional exogeneity (given a more comprehensive set of controls). For example, one could consider variation in the same treatment-variable and look at a larger selection of counties from across the US for which a larger set of controls is available (e.g., counties with more than 65,000 people for which a wide array of yearly controls are available from the ACS).

Of course, all non-experimental studies will typically be biased. When trying to study the causal effect of a treatment in a non-experimental way, we can rarely eliminate biases altogether. We can only try to reduce endogeneity biases.

My point is that, in trying to do so (reduce biases), there is often a trade-off between a more quasi-experimental setting with fewer controls, and a more observational one with more control. In the first case, causal claims rely mainly on studying situations that are as close as possible to a random experiment (because once that fails, we don't have many controls to pick up the tab and mitigate endogeneities). In the second, causal claims rely more on getting as close as possible to conditional exogeneity. Each approach is exposed to different risks of biases. In the first case, biases come more predominantly from the treatment not being exactly exogenous. In the second, they come more predominantly from omitted variables.

These days, it seems like econ is choosing the first side of the trade-off: Your chances at publishing a quasi-experimental study with even a few extra controls tend to be much larger than your chances at publishing a more observational study with a wide array of controls.

My question is: Has anyone studied this trade-off in a more or less systematic way, e.g., through Monte Carlo simulations? Do we have any sense, even in a particular case, of how the two sources of bias trade-off? For example, assume a particular distribution for the data and suppose that you plan on running a Diff-in-Diff based on a common trend assumption, but with no access to extra controls to ensure conditional common trends. As anyone looked into how much violation of the common trend assumption (i.e., how much correlation between the dependent variable and some controls) makes the Diff-in-Diff estimates more biased than a simple OLS that incorporates even a subset of the controls?

  • 1
    $\begingroup$ "...conditional exogeneity (given the more comprehensive set of controls" is a rather strong condition, arguably one that's usually unobtainable for observational data, no? Often, exogeneity cannot be achieved by simply adding more controls because, for example, behavioral bias inherent in the data generating process. $\endgroup$ – Michael Jun 18 '20 at 21:02
  • $\begingroup$ @Michael: Agreed. It's the same for quasi-experimental studies though. The reason they are called quasi-experimental is that they are not real random experiments. They can approach exogeneity but not achieve it to the same extent as actual random experiments. So all non-experimental studies, whether purely observational or quasi-experimental, will typically be biased. The question is not whether they are biased, but how much? I guess I should have written "put more of the weight of causal inference on getting closer to conditional exogeneity". I'll make an edit. $\endgroup$ – FZS Jun 18 '20 at 23:11
  • $\begingroup$ @Michael: Could you tell me more about what you mean by "Often, exogeneity cannot be achieved by simply adding more controls because, for example, behavioral bias inherent in the data generating process."? $\endgroup$ – FZS Jun 18 '20 at 23:12
  • $\begingroup$ There's no control for, say, people's tendency's to inflation their height/wage/etc. Or, in the context of the Lucas critique, how do you control for behavioral parameters that vary with respect to the policy in question? In such case, it seems that you would need quasi-experimental techniques, e.g. DID/RDD/what have you, to plausibly argue for exogeneity. $\endgroup$ – Michael Jun 18 '20 at 23:48
  • $\begingroup$ Thanks for the clarification. "people's tendency's to inflation their height/wage/etc." Are you talking about measurement error? If so, it affects both observational and quasi-experimental methods. If present, the study will be biased regardless of its "type". "behavioral parameters that vary with respect to the policy". Makes sense. I guess you could nitpick and say "what if you have a control for the behavioral parameter itself"? Not completely impossible for parameters like "risk-aversion". But I take your general point. $\endgroup$ – FZS Jun 19 '20 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.