Analysing data from a randomised experiment where independence fails

Question

Suppose that I sell ice cream and want to know whether charging \$1 or \$2 raises more revenue. To do this, I conduct a randomised price experiment. Specifically, on every day, I flip a coin and use that coin to determine whether the day’s price should be \$1 or \$2.

My question is how to analyse the results of this experiment. The most conservative approach would be to simply treat each day as a datapoint and compare revenue on \$1 days and \$2 days. That’s fine, but it makes the experiment very expensive to run. Indeed, to detect reasonable differences with conventional levels of power, one would need to run the experiment for at least several hundred days.

The least conservative approach would be to view the data as a series of (individual) customer level interactions. Specifically, one would pool all \$1 offers, pool all \$2 offers, and compare the share of offers that are accepted. One would then have a very large sample size, even if a small number of days. (For example, imagine that you only run the experiment for a couple of days, but that you offer an ice cream to thousands of customers on every day). One might then compare the two groups using a t-test (or equivalently, using linear regression). This will underestimate standard errors, however, since it neglects the fact that observations on a given day will be correlated (e.g. due to weather effects).

I believe that a ‘middle approach’ is possible and desirable. For example, one might analyse data on the customer level, but cluster standard errors at the day level. Am I right about this? (And is it actually different from what I calling the 'most conservative' approach?) Also, is there a textbook where I can learn more about these issues?

Answer 1

In terms of a textbook, The Handbook of Experimental Economics, John Kagel and Alvin E. Roth, editors, Princeton University Press, 1995.

I'm not sure the customer-based approach is very feasible. Is the idea that many customers approach the ice cream stand each day, some buy, some don't buy, and your outcome is a binary variable for buying, with a regressor being the price on the day? It's not crazy, but I think in practice it wouldn't work. Customers will likely know the price on a day and only approach the stand if they are willing to buy. Thus, the outcome will almost always be 1.

This idea might be more feasible in an online setting in which the price shown on the website is randomized, and then some customers buy or not. Many online retailers do this to learn the demand curve they face.

In general, estimating demand curves is extremely hard. You can look up the Berry, Levinsohn, and Pakes (1995) method for the primary tool used by economists in non-experimental methods.

Answer 2

A way to find an answer to your question is to make identification assumptions clear. Here, you need the SUTVA to hold (and that might be problematic in your case):

Consumers' decisions to buy an ice cream today are independent of the price of the ice cream in the past. In other words, you don't have spillover effects from one day to another.

Then, if your treatment is perfectly randomised, the price on a given day $p_t$ should be independent of the weather on that day $w_t$ (or any other covariates you can think of). As such, when looking at the effect of price on the demand for ice cream (let's call it $D_t$), you can simply run the following OLS:

$$D_t = \alpha + \beta \cdot p_t + \epsilon_t$$

But for that, you will need perfect randomisation (i.e., price is independent of any covariates you might think of) and SUTVA to hold (i.e., only price in period $t$ affects decisions of period $t$).

Answer 3

I have just realised that this is just a cluster randomised trial (CRT). Usually, clusters are geographical units that are treated or not (e.g. different villages); here, however, the clusters are time units (days).

As I expected, the standard analysis of a CRT yields results between those from the 'most conservative' and 'least conservative' approaches discussed in the question. To illustrate, here is an example dataset. In the example, there are 4 days, evenly divided between the low price (\$1) and high price (\$2). Assume that, on every day, 12 customers approach the stall (without knowing the price in advance).

Day	Price	Sales
1	1	10
2	2	5
3	1	12
4	2	7

On average, sales are 11 on low price days and 6 on high price days. How do we compare these numbers statistically?

• If we simply regress sales on price, which implicitly treats all units as independent, then we obtain a p-value of 0.001. This is the least conservative approach and seems invalid: for example, weather effects will create correlations within a day.
• If we analyse data at the day level, then we have just 4 observations and obtain a p-value of 0.072. This is the most conservative approach.
• If regress sales on price, but cluster at the day level, then we get a p-value of 0.023. In case others are interested, I did this using the STATA command regress sales price, vce(cluster day).

Notice that the p-value from the (?) correct approach, namely a clustered regression, lies between those obtained from the other two approaches.