Suppose that I estimate a (frequentist) confidence interval for the sample mean of a variable $X$, say at the 95% level. Suppose I also estimate a 95% confidence for the sample mean of a variable $Y$.

Claim: we cannot reject the hypothesis that $X$ and $Y$ have the same mean at the 5% level if and only if these confidence intervals overlap.

Is this true? I have a vague recollection that the answer is 'no'.


Yes your vague recollection is correct. You can verify it with a trivial example.

Lets take $\bar{X}=10$, with $\sigma(X)=2$ and $n=30$. So the $95\%$ confidence interval (using student's t distribution) is approximately given by $10 ± 0.75$.

Now suppose $\bar{Y}=11$ with $\sigma(Y)=1$ and $n=30$ so the $95\%$ confidence interval is given by approximately $11 ± 0.37$.

Confidence intervals above clearly overlap since for $\bar{X}$ the upper bound is 10.75, but for $\bar{Y}$ the lower bound is $10.63$ but when we actually do the t-test for difference in their means we get:

$$ t=\frac{10-11}{\sqrt{\left(\frac{2^2}{30}+\frac{1^2}{30}\right)}} \\ \approx -2.5 $$

Hence in this case we clearly can reject the null hypothesis that the means are same even at $5\%$ (in fact the $t$-stat is very close to being rejected at $1\%$ even) despite that their confidence levels happen to overlap.

  • $\begingroup$ Thanks, very helpful! But I remain a bit confused. Say $\bar{Y} > \bar{X}$ as in your example. Then the intervals overlap (using the CLT approximation) iff $\bar{Y} - 1.96SE(\bar{Y}) > \bar{X} + 1.96SE(\bar{X})$, or $\bar{Y} - \bar{X} > 1.96[SE(\bar{X}) + SE(\bar{Y})]$. Meanwhile, the means are `statistically different' iff $\bar{Y} - \bar{X} - 1.96SE(\bar{X} - \bar{Y}) = \bar{Y} - \bar{X} - 1.96[SE(\bar{X}) + SE(\bar{Y})] > 0$: same inequality! So what am I missing? $\endgroup$
    – afreelunch
    Feb 11 at 13:38
  • $\begingroup$ Perhaps I am wrong that the standard error of the difference in means is the sum of the standard errors of the means? $\endgroup$
    – afreelunch
    Feb 11 at 13:40
  • $\begingroup$ @afreelunch yes that does not hold since generally (save for special cases) $se(X)-se(Y)\neq se(Y-X)$ (also inside se() you will not add bar - bar $\bar{X}$ stands for mean of random variable $X$ whether se(X) is standard error of random variable X). $\endgroup$
    – 1muflon1
    Feb 11 at 13:51
  • $\begingroup$ By 'standard error', I mean standard deviation of the sample mean; not the standard deviation of the random variable (I think this is the usual terminology?) $\endgroup$
    – afreelunch
    Feb 11 at 16:16
  • 1
    $\begingroup$ But I think I have found my mistake: $$SE[\bar{X} - \bar{Y}] = Var[\bar{X} - \bar{Y}]^{0.5} = [Var[\bar{X}] + Var[\bar{Y}]]^{0.5} = [SE[\bar{X}]^2 + SE[\bar{Y}]^2]^{0.5}$$ which does not generally equal $$SE[\bar{X}] + SE[\bar{Y}]$$ So the standard error of the difference in means is not equal to the sum of standard errors. $\endgroup$
    – afreelunch
    Feb 11 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.