Take the 2-minute tour ×
Economics Stack Exchange is a question and answer site for professional and academic economists and analysts. It's 100% free, no registration required.

P-value hacking is the "art" of looking at different outcomes and specifications until you get a "false positive", i.e. a p value under, say, 0.05, which only noise and not true under the data generating process.

Say I have a treated group with size $N$ and a control group with size $M$, $K$ outcome variables, and am targeting a p-value of $p$: How can I compute the ex-ante probability of getting at least one false positive significant result significant under $p$?

You can assume that the $K$ characteristics are independently and normal distributed, and if it simplifies a lot, that $M=N$.

share|improve this question
    
Full disclosure: I am impressed by a fairly interesting result where $M+N = 50$. I'd like to get a rough approximation of how likely their interesting result stems from too many variables of interest. –  FooBar May 29 at 18:33
    
What exactly is your null hypothesis? That the average of a given characteristic is the same for both groups? (And this is repeated for all $K$ variables.) I am not sure but I think you would also have to say something about the type of the underlying probability distribution. –  denesp May 29 at 19:30
    
A possibly interesting and relevant article. A quote from the article, "Fujii’s subsequent dismissal was soon followed by a flood of damning evidence about his work. On 8 March, Anaesthesia published an analysis by John Carlisle, a consultant anaesthetist at Torbay Hospital in Torquay, UK, finding that 168 of Fujii’s papers had results with 'likelihoods that are infinitesimally small.' " Summary: A guy used statistics to show multiple of Yoshitaka Fujii's results were bogus –  cc7768 May 29 at 20:18
    
Off topic => stats.stackexchange.com –  André Peseur May 29 at 20:34
2  
Foobar, yeah that's why I said possibly relevant haha -- It isn't quite directly related, but your question reminded me of it. Your article seems a little more related :) @AndréPeseur, I think there is going to be some overlap in topics between our website and cross-validated. I'm of the opinion that econometrics should be on-topic here -- Not a SE pro or anything though. Maybe start a meta post to discuss it further if you disagree. –  cc7768 May 29 at 20:41

3 Answers 3

Under the assumption of i.i.d. Normal characteristics, the situation described is taken care by separate Welch's t-tests that account for possibly different sample sizes and different variances. Denote the statistics of these tests $t_j, j=1,...,K$. The p-value associated with each is

$$p_j = \Pr\big(|t_j|\geq t(\alpha)\mid H_0\big) $$

where $H_0$ is the hypothesis that the populations means between treated and controlled group are equal, and $t$ depends on the significance level $1-\alpha$.

We can write the probability in terms of the corresponding cumulative distribution function,

$$\Pr\big(|t_j|\geq t(\alpha)\mid H_0\big) = 1 - F(|t_j|)$$

Therefore

$$p_j = 1 - F(|t_j|) \implies 1-p_j = F(|t_j|)$$

If we contemplate the situation a priori, before even looking at the data, then the p-values lie in the future and can be modelled as random variables. Viewed as a random variable, the probability integral transform tells us that $1-p_j$ follows a $U(0,1)$ Uniform distribution, and by the properties of this distribution so does $p_j$.

Collecting all $p_j$, we have a sample of size $K$ of independent $U(0,1)$ uniforms. The probability that at least one of them is smaller that a specific value, say $p^*$, is equal to the probability that the minimum of them is lower than this threshold. This can be understood as follows:

$$\Pr\Big (\text {At least one $p_j \leq p^*$} \Big) = \Pr\Big (\text {Not all $p_j > p^*$} \Big) $$

$$ = 1-\Pr\Big (\text {All $p_j > p^*$} \Big) = 1- \prod_{j=1}^K \Pr\Big ( p_j > p^* \Big)$$

due to independence, and so, since they are identically distributed,

$$\Pr\Big (\text {At least one $p_j \leq p^*$} \Big) = 1- \left [1-\Pr\Big ( p \leq p^* \Big)\right]^K = 1 - \left [1-F_U \big(p^* \big)\right]^K$$

But this is the cumulative distribution function of the minimum of $K$ i.i.d random variables.

Denote this minimum $p_{(1)}$.

The CDF of the minimum of $K$ independent $U(0,1)$ variables is

$$F_{p_{(1)}}(p_{(1)}) = 1 - \big [1-p_{(1)}\big]^K$$

We want the probability

$$ \Pr(p_{(1)} \leq p^*) = 1- \big [1-p^*\big]^K$$

Indicative values:

enter image description here

share|improve this answer
    
"Welch's t-tests that account for possibly different sample sizes and different variances". I had only time to skim the reply so far, but I can't find where the sample sizes $M$, $N$ enter the picture. How does the last table vary with the sample size? –  FooBar 2 days ago
1  
It doesn't. As random variables, the p-values are Uniforms (0,1) whatever the other aspects of the situation. The only thing that matters is the size of $K$. –  Alecos Papadopoulos 2 days ago
    
That's strange. This link claims something else: "If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people.". What are the underlying assumptions here that lead to there different result? –  FooBar 2 days ago
1  
The crucial part of the statement is "a large number of things", i.e. a large $K$. Go back to the link and read just below the sub-chapter "The Hook". You will see the same formula as in my answer, and a percentage consistent with my table. –  Alecos Papadopoulos 2 days ago
    
I see. So his formulation "large number of things about a small number of people" implies the wrong thing, it should be without "small number of people" –  FooBar 2 days ago

I agree with @AlecosPapadopoulos we want something like: $$ \Pr(p_{(1)} \leq p^*) = 1- \big [1-p^*\big]^K$$ But I don't see how $n$ and $M$ couldn't enter into the proper test statistic. For example, if the underlying data is normally distributed i.i.d. data then $N$ and $M$ do matter.

Consider that noise mean $\mu$ and variance $\sigma$, which, by assumption is the same for the control and "treated" group. The mean of the treated group with size N will be distributed $N(\mu, \sigma^2 / n)$ and $N(\mu, \sigma^2 / M)$ for the control. So the difference in means will be distributed $$N(0, \sigma^2 / n + \sigma^2 / m)$$

But you won't know $\sigma$ or $\mu$, so we'll have to estimate it with $X_1$, $X_2$, and $s_{X_1X_2}$, and use a t-test. This setup gives a t-tatistic like this: $$t = \frac{\bar {X}_1 - \bar{X}_2}{s_{X_1 X_2} \cdot \sqrt{\frac{1}{n}+\frac{1}{m}}}$$ where $$s_{X_1X_2} = \sqrt{\frac{(n-1)s_{X_1}^2+(m-1)s_{X_2}^2}{n+m}}.$$ SRC: Student's t-test on Wikipedia

The unpaired sample t-test for this difference in means has degrees of freedom $N-M-2$. Therefore the rejection region should depend on both n and m, both in what critical value of the test to use through the degrees of freedom of that test and the test statistic calculation itself.

share|improve this answer
1  
Certainly the sample sizes enter the test-statistic, but this does not affect the specific probability the OP asks about (i.e. the a priori probability that we would obtain at least one p-value lower than a given threshold). –  Alecos Papadopoulos yesterday
    
A properly specified test would incorporate the multiple comparisons just as it would sample size and degrees of freedom - - right? So is this a question about using the wrong test statistic along only the multiple comparison dimension but properly specified on the individual comparison dimension? Because the presence of m and n in the test statistic is exactly because a small sample size is more likely to have a large difference under the null hypothesis. –  BKay yesterday
1  
The crucial point here is that a priori, a) a p-value has a marginal U(0,1) distribution, irrespective of anything else (sample size or whatever). This is a general result holding in any kind of situation. This should be intuitive: what would happen if the p-value had a priori a non-uniform distribution? CONTD –  Alecos Papadopoulos yesterday
1  
CONTD It would mean that some of its values would be more likely than others, again irrespective of the specifics of the situation. But this would invalidate the whole testing procedure, exactly because it would imply that "it doesn't matter what you test, what is the sample size etc - this value for the p-value is more probable than others". –  Alecos Papadopoulos yesterday
    
Thank both of you for the extended discussion, I need to learn more about this and I feel this should definitively be part of a graduate studies curriculum. –  FooBar yesterday

There is one thing perhaps worth adding to the excellent answers above, which is there is essentially a meta-numbers game going on as well. Lets say that 20 scientists all do the same set of experiments looking for something possibly weakly correlated like "does chocolate cause heart attacks", and will accept the p value < 0.05 significant which frankly they shouldn't. The cumulative probability is that one scientist will get a significant finding, which is the one experiment that will get published, since negative results rarely get accepted. There is then a 100% chance that that finding will get picked up by the Bild Zeitungs of this world and mis-reported.

Unfortunately, because we don't report the absence of findings, we're essentially engaged in a planet wide exercise in reporting all the experiments that get lucky - in the wrong sense of the word.

For subjects with a strong theoretical basis, good experimental design provides some protection against this - for subjects which are pre-dominantly having to work with observational data, and try to work out the theory - like economics - it's a major issue.

share|improve this answer
    
This is a very good remark. The problem plagues findings and meta findings based on earlier papers as well. However I think it is perhaps out of sync with this particular question, as the thought experiment seems to about an individual hack scientist measuring his chances? –  denesp 9 hours ago
    
I don't disagree, however since Foobar was asking in the context of a paper he was looking at, I thought it wouldn't hurt to throw in the worst case analysis. –  Lumi 8 hours ago

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.