Suppose that I sell ice cream and want to know whether charging \$1 or \$2 raises more revenue. To do this, I conduct a randomised price experiment. Specifically, on every day, I flip a coin and use that coin to determine whether the day’s price should be \$1 or \$2.
My question is how to analyse the results of this experiment. The most conservative approach would be to simply treat each day as a datapoint and compare revenue on \$1 days and \$2 days. That’s fine, but it makes the experiment very expensive to run. Indeed, to detect reasonable differences with conventional levels of power, one would need to run the experiment for at least several hundred days.
The least conservative approach would be to view the data as a series of (individual) customer level interactions. Specifically, one would pool all \$1 offers, pool all \$2 offers, and compare the share of offers that are accepted. One would then have a very large sample size, even if a small number of days. (For example, imagine that you only run the experiment for a couple of days, but that you offer an ice cream to thousands of customers on every day). One might then compare the two groups using a t-test (or equivalently, using linear regression). This will underestimate standard errors, however, since it neglects the fact that observations on a given day will be correlated (e.g. due to weather effects).
I believe that a ‘middle approach’ is possible and desirable. For example, one might analyse data on the customer level, but cluster standard errors at the day level. Am I right about this? (And is it actually different from what I calling the 'most conservative' approach?) Also, is there a textbook where I can learn more about these issues?