I apologize in advance for a long and possibly misworded question. I am trying to get to something but I am not exactly sure how to explain it. Hopefully, you can bear with me and provide me with some guidance.
I am interested in the trade-off between observational studies with a large set of controls and quasi-experimental studies with fewer controls.
I often see quasi-experimental papers that (try to argue they) get some almost exogenous variation, but at the expense of having fewer data points and fewer control variables. For example, you often see people rely on a new state-law to induce changes in a treatment-variable. To analyze the treatment, they then use a yearly county-level panel-data over, say, a 10 years period including the adoption of the law. Looking within this single state over that a period may allow them to argue that the variation is almost exogenous. But it leaves them with few county-controls to mitigate for possible endogeneities (e.g., in the case of US counties, because yearly county data is only available from the ACS for counties of more than 65,000 people). That is, their claim to exogeneity relies mainly on the quasi-experimental nature of the treatment, and little on controls (conditional exogeneity).
Another option would have been to run a purely observational study with a larger set of controls and put more of the weight of causal inference on getting closer to conditional exogeneity (given a more comprehensive set of controls). For example, one could consider variation in the same treatment-variable and look at a larger selection of counties from across the US for which a larger set of controls is available (e.g., counties with more than 65,000 people for which a wide array of yearly controls are available from the ACS).
Of course, all non-experimental studies will typically be biased. When trying to study the causal effect of a treatment in a non-experimental way, we can rarely eliminate biases altogether. We can only try to reduce endogeneity biases.
My point is that, in trying to do so (reduce biases), there is often a trade-off between a more quasi-experimental setting with fewer controls, and a more observational one with more control. In the first case, causal claims rely mainly on studying situations that are as close as possible to a random experiment (because once that fails, we don't have many controls to pick up the tab and mitigate endogeneities). In the second, causal claims rely more on getting as close as possible to conditional exogeneity. Each approach is exposed to different risks of biases. In the first case, biases come more predominantly from the treatment not being exactly exogenous. In the second, they come more predominantly from omitted variables.
These days, it seems like econ is choosing the first side of the trade-off: Your chances at publishing a quasi-experimental study with even a few extra controls tend to be much larger than your chances at publishing a more observational study with a wide array of controls.
My question is: Has anyone studied this trade-off in a more or less systematic way, e.g., through Monte Carlo simulations? Do we have any sense, even in a particular case, of how the two sources of bias trade-off? For example, assume a particular distribution for the data and suppose that you plan on running a Diff-in-Diff based on a common trend assumption, but with no access to extra controls to ensure conditional common trends. As anyone looked into how much violation of the common trend assumption (i.e., how much correlation between the dependent variable and some controls) makes the Diff-in-Diff estimates more biased than a simple OLS that incorporates even a subset of the controls?
