# P-value hacking

P-value hacking is the "art" of looking at different outcomes and specifications until you get a "false positive", i.e. a p value under, say, 0.05, which only noise and not true under the data generating process.

Say I have a treated group with size $N$ and a control group with size $M$, $K$ outcome variables, and am targeting a p-value of $p$: How can I compute the ex-ante probability of getting at least one false positive significant result significant under $p$?

You can assume that the $K$ characteristics are independently and normal distributed, and if it simplifies a lot, that $M=N$.

-
Full disclosure: I am impressed by a fairly interesting result where $M+N = 50$. I'd like to get a rough approximation of how likely their interesting result stems from too many variables of interest. –  FooBar May 29 at 18:33
What exactly is your null hypothesis? That the average of a given characteristic is the same for both groups? (And this is repeated for all $K$ variables.) I am not sure but I think you would also have to say something about the type of the underlying probability distribution. –  denesp May 29 at 19:30
A possibly interesting and relevant article. A quote from the article, "Fujii’s subsequent dismissal was soon followed by a flood of damning evidence about his work. On 8 March, Anaesthesia published an analysis by John Carlisle, a consultant anaesthetist at Torbay Hospital in Torquay, UK, finding that 168 of Fujii’s papers had results with 'likelihoods that are infinitesimally small.' " Summary: A guy used statistics to show multiple of Yoshitaka Fujii's results were bogus –  cc7768 May 29 at 20:18
Off topic => stats.stackexchange.com –  André Peseur May 29 at 20:34
Foobar, yeah that's why I said possibly relevant haha -- It isn't quite directly related, but your question reminded me of it. Your article seems a little more related :) @AndréPeseur, I think there is going to be some overlap in topics between our website and cross-validated. I'm of the opinion that econometrics should be on-topic here -- Not a SE pro or anything though. Maybe start a meta post to discuss it further if you disagree. –  cc7768 May 29 at 20:41

Under the assumption of i.i.d. Normal characteristics, the situation described is taken care by separate Welch's t-tests that account for possibly different sample sizes and different variances. Denote the statistics of these tests $t_j, j=1,...,K$. The p-value associated with each is

$$p_j = \Pr\big(|t_j|\geq t(\alpha)\mid H_0\big)$$

where $H_0$ is the hypothesis that the populations means between treated and controlled group are equal, and $t$ depends on the significance level $1-\alpha$.

We can write the probability in terms of the corresponding cumulative distribution function,

$$\Pr\big(|t_j|\geq t(\alpha)\mid H_0\big) = 1 - F(|t_j|)$$

Therefore

$$p_j = 1 - F(|t_j|) \implies 1-p_j = F(|t_j|)$$

If we contemplate the situation a priori, before even looking at the data, then the p-values lie in the future and can be modelled as random variables. Viewed as a random variable, the probability integral transform tells us that $1-p_j$ follows a $U(0,1)$ Uniform distribution, and by the properties of this distribution so does $p_j$.

Collecting all $p_j$, we have a sample of size $K$ of independent $U(0,1)$ uniforms. The probability that at least one of them is smaller that a specific value, say $p^*$, is equal to the probability that the minimum of them is lower than this threshold. This can be understood as follows:

$$\Pr\Big (\text {At least one p_j \leq p^*} \Big) = \Pr\Big (\text {Not all p_j > p^*} \Big)$$

$$= 1-\Pr\Big (\text {All p_j > p^*} \Big) = 1- \prod_{j=1}^K \Pr\Big ( p_j > p^* \Big)$$

due to independence, and so, since they are identically distributed,

$$\Pr\Big (\text {At least one p_j \leq p^*} \Big) = 1- \left [1-\Pr\Big ( p \leq p^* \Big)\right]^K = 1 - \left [1-F_U \big(p^* \big)\right]^K$$

But this is the cumulative distribution function of the minimum of $K$ i.i.d random variables.

Denote this minimum $p_{(1)}$.

The CDF of the minimum of $K$ independent $U(0,1)$ variables is

$$F_{p_{(1)}}(p_{(1)}) = 1 - \big [1-p_{(1)}\big]^K$$

We want the probability

$$\Pr(p_{(1)} \leq p^*) = 1- \big [1-p^*\big]^K$$

Indicative values:

-
"Welch's t-tests that account for possibly different sample sizes and different variances". I had only time to skim the reply so far, but I can't find where the sample sizes $M$, $N$ enter the picture. How does the last table vary with the sample size? –  FooBar 2 days ago
It doesn't. As random variables, the p-values are Uniforms (0,1) whatever the other aspects of the situation. The only thing that matters is the size of $K$. –  Alecos Papadopoulos 2 days ago
That's strange. This link claims something else: "If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people.". What are the underlying assumptions here that lead to there different result? –  FooBar 2 days ago
The crucial part of the statement is "a large number of things", i.e. a large $K$. Go back to the link and read just below the sub-chapter "The Hook". You will see the same formula as in my answer, and a percentage consistent with my table. –  Alecos Papadopoulos 2 days ago
I see. So his formulation "large number of things about a small number of people" implies the wrong thing, it should be without "small number of people" –  FooBar 2 days ago

I agree with @AlecosPapadopoulos we want something like: $$\Pr(p_{(1)} \leq p^*) = 1- \big [1-p^*\big]^K$$ But I don't see how $n$ and $M$ couldn't enter into the proper test statistic. For example, if the underlying data is normally distributed i.i.d. data then $N$ and $M$ do matter.

Consider that noise mean $\mu$ and variance $\sigma$, which, by assumption is the same for the control and "treated" group. The mean of the treated group with size N will be distributed $N(\mu, \sigma^2 / n)$ and $N(\mu, \sigma^2 / M)$ for the control. So the difference in means will be distributed $$N(0, \sigma^2 / n + \sigma^2 / m)$$

But you won't know $\sigma$ or $\mu$, so we'll have to estimate it with $X_1$, $X_2$, and $s_{X_1X_2}$, and use a t-test. This setup gives a t-tatistic like this: $$t = \frac{\bar {X}_1 - \bar{X}_2}{s_{X_1 X_2} \cdot \sqrt{\frac{1}{n}+\frac{1}{m}}}$$ where $$s_{X_1X_2} = \sqrt{\frac{(n-1)s_{X_1}^2+(m-1)s_{X_2}^2}{n+m}}.$$ SRC: Student's t-test on Wikipedia

The unpaired sample t-test for this difference in means has degrees of freedom $N-M-2$. Therefore the rejection region should depend on both n and m, both in what critical value of the test to use through the degrees of freedom of that test and the test statistic calculation itself.

-
Certainly the sample sizes enter the test-statistic, but this does not affect the specific probability the OP asks about (i.e. the a priori probability that we would obtain at least one p-value lower than a given threshold). –  Alecos Papadopoulos yesterday
A properly specified test would incorporate the multiple comparisons just as it would sample size and degrees of freedom - - right? So is this a question about using the wrong test statistic along only the multiple comparison dimension but properly specified on the individual comparison dimension? Because the presence of m and n in the test statistic is exactly because a small sample size is more likely to have a large difference under the null hypothesis. –  BKay yesterday
The crucial point here is that a priori, a) a p-value has a marginal U(0,1) distribution, irrespective of anything else (sample size or whatever). This is a general result holding in any kind of situation. This should be intuitive: what would happen if the p-value had a priori a non-uniform distribution? CONTD –  Alecos Papadopoulos yesterday
CONTD It would mean that some of its values would be more likely than others, again irrespective of the specifics of the situation. But this would invalidate the whole testing procedure, exactly because it would imply that "it doesn't matter what you test, what is the sample size etc - this value for the p-value is more probable than others". –  Alecos Papadopoulos yesterday