Did previous researchers fail to detect the hot hand simply because of a statistical fallacy?

Question

Many basketball fans/players believe that having made several shots in a row, the next shot is more likely to go in. This is sometimes called the hot hand.

Starting (I think) with Gilovich, Mallone, and Tversky (1985), it was "shown" that this was in fact a fallacy. Even if several shots in a row have gone in, the next shot is no more likely to go in than your average shooting percentage would dictate.

Miller and Sanjurjo (2015) argue that the hot hand does in fact exist and previous researchers had simply fallen prey to a fairly basic statistical fallacy. Their argument is something like this:

Flip a coin four times. Compute the probability that H follows H. To give a few examples: HHTT would have probability 1/2, HTHT would have probability 0/2, TTHH would have probability 0/1 1/1, and both TTTT and TTTH would be N.A.

Miller and Sanjurjo's punchline is that the expected value of this probability is not 0.5, but ≈0.4. And the error made by previous researchers was to incorrectly assume that the expected value of this probability is 0.5. So if for example these previous researchers conducted the above coin-flipping experiment and found the average probability to be say 0.497, they incorrectly concluded that there was no evidence of a hot hand (not significantly different from 0.5), when in fact there was very strong evidence of a hot hand (significantly different from 0.4).

My question is this: Are Miller and Sanjurjo correct that previous researchers failed to detect the hot hand simply because of this mistake? I have only skimmed one or two papers on this so I wanted to get some confirmation from someone here who might know this literature better. This seems like a surprisingly silly error to have persisted for three decades or more.

Answer 1

(This answer was completely rewritten for greater clarity and readability in July 2017.)

Flip a coin 100 times in a row.

Examine the flip immediately after a streak of three tails. Let $\hat{p}(H|3T)$ be the proportion of coin flips after each streak of three tails in a row that are heads. Similarly, let $\hat{p}(H|3H)$ be the proportion of coin flips after each streak of three heads in a row that are heads. (Example at bottom of this answer.)

Let $x:=\hat{p}(H|3H)-\hat{p}(H|3T)$.

If the coin-flips are i.i.d., then "obviously", across many sequences of 100 coin-flips,

(1) $x>0$ is expected to happen as often as $x<0$.

(2) $E(X)=0$.

We generate a million sequences of 100 coin-flips and get the following two results:

(I) $x>0$ happens roughly as often as as $x<0$.

(II) $\bar{x} \approx 0$ ($\bar{x}$ is the average of $x$ across the million sequences).

And so we conclude that the coin-flips are indeed i.i.d. and there is no evidence of a hot hand. This is what GVT (1985) did (but with basketball shots in place of coin-flips). And this is how they concluded that the hot hand does not exist.

---

Punchline: Shockingly, (1) and (2) are incorrect. If the coin-flips are i.i.d., then it should instead be that

(1-corrected) $x>0$ occurs only about 37% of the time, while $x<0$ occurs about 60% of the time. (In the remaining 3% of the time, either $x=0$ or $x$ is undefined — either because there was no streak of 3H or no streak of 3T in the 100 flips.)

(2-corrected) $E(X) \approx -0.08$.

The intuition (or counter-intuition) involved is similar to that in several other famous probability puzzles: the Monty Hall problem, the two-boys problem, and the principle of restricted choice (in the card game bridge). This answer is already long enough and so I'll skip the explanation of this intuition.

And so the very results (I) and (II) obtained by GVT (1985) are actually strong evidence in favor of the hot hand. This is what Miller and Sanjurjo (2015) showed.

---

Further analysis of GVT's Table 4.

Many (e.g. @scerwin below) have — without bothering to read GVT (1985) — expressed disbelief that any "trained statistician would ever" take an average of averages in this context.

But that is exactly what GVT (1985) did in their Table 4. See their Table 4, columns 2-4 and 5-6, bottom row. They find that averaged across the 26 players,

$\hat{p}(H|1M) \approx 0.47$ and $\hat{p}(H|1H) \approx 0.48$,

$\hat{p}(H|2M) \approx 0.47$ and $\hat{p}(H|2H) \approx 0.49$,

$\hat{p}(H|3M) \approx 0.45$ and $\hat{p}(H|3H) \approx 0.49$.

Actually it is the case that for each $k=1,2,3$, the averaged $\hat{p}(H|kH)>\hat{p}(H|kM)$. But GVT's argument seems to be that these are not statistically significant and so these are not evidence in favor of the hot hand. OK fair enough.

But if instead of taking the average of averages (a move considered unbelievably stupid by some), we redo their analysis and aggregate across the 26 players (100 shots for each, with some exceptions), we get the following table of weighted averages.

Any                     1175/2515 = 0.4672

3 misses in a row       161/400 = 0.4025
3 hits in a row         179/313 = 0.5719

2 misses in a row       315/719 = 0.4381
2 hits in a row         316/581 = 0.5439        

1 miss in a row         592/1317 = 0.4495
1 hit in a row          581/1150 = 0.5052

The table says, for example, that a total of 2,515 shots were taken by the 26 players, of which 1,175 or 46.72% were made.

And of the 400 instances where a player missed 3 in a row, 161 or 40.25% were immediately followed by a hit. And of the 313 instances where a player hit 3 in a row, 179 or 57.19% were immediately followed by a hit.

The above weighted averages seem to be strong evidence in favor of the hot hand.

Bear in mind that the shooting experiment was set up so that each player was shooting from where it had been determined he/she could make roughly 50% of his/her shots.

(Note: "Strangely" enough, in Table 1 for a very similar analysis with the Sixers' in-game shooting, GVT instead present the weighted averages. So why didn't they do the same for Table 4? My guess is that they certainly did calculate the weighted averages for Table 4 — the numbers I present above, didn't like what they saw, and chose to suppress them. This sort of behavior is unfortunately par for the course in academia.)

---

Example: Say we have the sequence $HHHTTTHHHHH…H$ (only flips #4-#6 are tails, the remaining 97 flips are all heads). Then $\hat{p}(H|3T)=1/1=1$ because there is only 1 streak of three tails and the flip immediately after that streak is heads.

And $\hat{p}(H|3H)=91/92 \approx 0.989$ because there are 92 streaks of three heads and for 91 of those 92 streaks, the flip immediately after is heads.

---

P.S. GVT's (1985) Table 4 contains several errors. I spotted at least two rounding errors. And also for player 10, the parenthetical values in columns 4 and 6 do not add up to one less than that in column 5 (contrary to the note at the bottom). I contacted Gilovich (Tversky is dead and Vallone I am not sure), but unfortunately he no longer has the original sequences of hits and misses. Table 4 is all we have.

Answer 2

Neither of the two papers are clear enough as regards their applications of Statistics, so in this answer I will attempt a clarification.

Gilovich, Mallone, and Tversky (1985) in their Abstract define the "Hot-Hand effect" as follows:

"Basketball players and fans alike tend to believe that a player’s chance of hitting a shot are greater following a hit than following a miss on the previous shot."

"Previous shot" is then extended to previous "one, two or three" shots. Denoting a series of $k$ sequential Hits by $H_k$ and a series of $k$ sequential misses by $M_k$, the presence of the Hot-Hand effect is defined as

$$P(H \mid H_k) > P(H\mid M_k),\;\;\; k\geq 1 \tag{1}$$

where for compactness, it is understood that the shot in question is the one immediately following the sequential hits or misses. These are theoretical conditional probabilities (i.e. constants), not conditional relative empirical frequencies.

How do the authors attempt to test the existence of the Hot-Hand Effect? They obtain empirical data, they calculate conditional relative empirical frequencies $\hat P(H \mid H_k) ,\; \hat P(H\mid M_k)$ (which are random variables) and they perform t-tests with null hypothesis (pp. 299-300)

$${\rm H_o:} P(H \mid H_k) - P(H\mid M_k) =0$$

Note by the way that this test is weaker than a test for independence of shots: these probabilities could be equal but still differing from the unconditional probability $P(H)$.

Naturally, the statistic used is $T\equiv \hat P(H \mid H_k) - \hat P(H\mid M_k)$. The authors find that the null is rejected at conventional significance levels, but in a direction against the Hot-Hand Hypothesis: the t-value is large enough but negative.

The question then is: is the test valid? First, in order for empirical frequencies to consistently estimate unknown probabilities, it must be the case that the sample is ergodic-stationary. It is, in this case (see the discussion on p.297). Then the other thing left to question is what is the distribution of the statistic $T$? Is it well approximated by a Student distribution for finite samples (since it is the critical values from the Student distribution that are used)? And for what sizes?

What Miller and Sanjurjo (2015) do is to argue (and apparently, prove) that the "exact" (finite-sample) distribution of $T$ has a non-negligible negative skew and a non-zero expected value,(see pp 18-19). If this is so, the use of the t-test can be misleading, at least for finite samples, event though it may remain valid asymptotically/for "large" samples.

Therefore, if there is a problem with the Gilovich et al. paper, it is not the definition of the Hot-Hand, it is not the formulation of the null-hypothesis, it is not the selection of the statistic to be used: it is the validity of the critical values used to execute the tests (and so of the implicit distributional assumption), if indeed the finite, small-sample distribution (under the null hypothesis) is visibly non-centered at zero and also asymmetric.

In such cases, what one does usually is to obtain by simulation special critical values in order to perform the test (remember for example the special critical values for the Dickey-Fuller test for a unit root). I failed to see such an approach in the Miller-Sanjurjo paper -instead, they perform "mean bias adjustment", and find that after this adjustment the conclusion from the test is reversed. I am not sure this is the way to go.

Nevertheless a rough simulation validates the Miller-Sanjurjo results as regards the distribution of the statistic. I simulated $200$ samples each of size $n=100$, of independent Bernoullis with $p=0.5$.
The empirical distribution of the statistic $T_3 = \hat P(H \mid H_3) - \hat P(H\mid M_3)$ has a sample mean of $-0.0807$ and a median of $-0.072$, with $62.5\%$ of the values being negative. The empirical histogram is

Answer 3

(Disclaimer: I don't know this literature.) It seems to me that Miller and Sanjurjo have a valid criticism of a particular statistical measure. I don't know if this should be considered to invalidate all prior work on the hot-hand effect, since they focus on only this particular measure.

The measure is

$$ M := P(\text{make shot }|\text{ made previous shot}) - P(\text{make shot }|\text{ miss previous shot}) $$ where $P(X)$ really means "fraction of times $X$ occurred".

Prior work, such as [Gilovich, Mallone, Tversky, 1985], claims that $M$ being close to zero or negative is evidence of a lack of the hot-hand effect. The implicit assumption is that $\mathbb{E} M > 0$ if there is a hot-hand effect and $\mathbb{E} M = 0$ otherwise. (See the subsection Analysis of Conditional Probabilities under Study 2.)

However, Miller and Sanjurjo point out that $\mathbb{E} M < 0$ if there is no hot-hand effect. Hence $M$ being close to zero does not suggest a lack of the hot-hand effect.

So again in summary, I have not actually answered your question on whether this paper invalidates prior work on the hot hand effect (which uses many different statistical measures), but it seems to me that the paper makes a valid point regarding this particular statistical measure. Specifically, for example, Gilovich, Mallone, Tversky uses non-positivity of $M$ as one piece of supporting evidence, and this paper shows the flaw in that argument.

Answer 4

In my view, Miller and Sanjurjo simply calculated the relative frequencies in Table 1 incorrectly. Their table is shown below with two new columns added, which count the number of subsequences HH and HT that occur within each sequence of 4 coin flips. To get the desired conditional probability p(H|H) one must sum these counts N(HH) and N(HT) and then divide as shown below. Doing this gives p(H|H)=0.5, as expected. For some reason, Miller and Sanjurjo first calculated the relative frequency for each sequence and then averaged over the sequences. That's just wrong.

Sequence     Subsequences       N(HH) N(HT)    p(H|H)
TTTT  ->  TT.. , .TT. , ..TT      0     0        -  
TTTH  ->  TT.. , .TT. , ..TH      0     0        -  
TTHT  ->  TT.. , .TH. , ..HT      0     1       0.0 
THTT  ->  TH.. , .HT. , ..TT      0     1       0.0 
HTTT  ->  HT.. , .TT. , ..TT      0     1       0.0 
TTHH  ->  TT.. , .TH. , ..HH      1     0       1.0 
THTH  ->  TH.. , .HT. , ..TH      0     1       0.0 
THHT  ->  TH.. , .HH. , ..HT      1     1       0.5 
HTTH  ->  HT.. , .TT. , ..TH      0     1       0.0 
HTHT  ->  HT.. , .TH. , ..HT      0     2       0.0 
HHTT  ->  HH.. , .HT. , ..TT      1     1       0.5 
THHH  ->  TH.. , .HH. , ..HH      2     0       1.0 
HTHH  ->  HT.. , .TH. , ..HH      1     1       0.5 
HHTH  ->  HH.. , .HT. , ..TH      1     1       0.5 
HHHT  ->  HH.. , .HH. , ..HT      2     1       0.66
HHHH  ->  HH.. , .HH. , ..HH      3     0       1.0 
                                 --    --       ----
                                 12    12       0.40
                            p(H|H)=N(HH)/N(H*)
                                  =12/(12+12)
                                  =0.5

Answer 5

I'm going to change a comment I made above to an answer, and claim the answer to the original question is that the original papers are correct. The authors of the 2015 paper throw out sequences which should logically be included in their analysis, as I describe in the comment, and therefore introduce a bias which supports their claims. The world works as it should.

Addendum in response to comment: We look at table 1 in the paper. We see we're throwing out 4 values from the last column, so to get the expected difference we only average over 12 of the 16 sequences. If we look at these probabilities as frequencies, and we say, for the first line TTTT, what is the frequency at which a head follows a head, then logically it always happens, and we should put a 1 in the p(H, H) column, not a dash. We do that for the other three sequences we threw out, and we conclude the expected value of the difference is 0, not -.33. We can't just throw out data like that, when there is a clear logical interpretation of the data.

Note that in order to make the drift vanish, we have to calculate the probabilities correctly, which isn't done in the paper. The probabilities in the table are claimed to be the "probability that a head follows a tail, in this given sequence of four tosses." And we see that for the row TTTH, we're supposed to believe that probability is 1/3. It's not. There are four tosses in the row, and one of the four tosses in that row is the event "a head follows a tail". The probability is 1/4. So calculate the probabilities correctly, and use all the rows, and you get the answer that's been accepted for 30 years.

Answer 6

In any observed sequence, the last conditional is "missing" in the sense that there is no value afterwards. The authors deal with this by simply disregarding cases where this happens, saying that they are undefined. If the series is short, this choice is going to have an obvious impact on calculations. Figure 1 is a nice illustration of this idea.