t-test result inconsistent with regression

Question

I have two batches of data and I want to check whether the response variable is different across the two batches. I first did a t-test and found that the means of the response variables in the two batches differ significantly. I then run a regression with an intercept and a Group 2 dummy. The coefficient of the Group 2 dummy is insignificant. This regression result seems to be inconsistent with the t-test result. Could anyone please explain this happens and provide a simple example? Thanks.

Answer 1

There are multiple ways to perform various test that can be defended, depending on the context, but they are not, necessarily, equivalent.

The typical way to calculate the standard errors in an OLS linear model assumes equal variances in the two groups (constant-variance error term for the general model). This is the default calculation in most software, and then the confidence intervals and p-values are derived from that standard error.

There is a variant of the t-test called Welch's t-test that assumes (or at least allows for) unequal variances in to two groups. It sounds like you might have used this calculation for what you are calling the "t-test". For example, this is the default calculation in the t.test function in R software.

While the usual t-test will agree with the usual OLS linear model coefficient test, Welch's test does not agree with the usual t-test.

It sounds like you used some variant of the usual t-test, such as Welch's test, to calculate the standard error and resulting t-statistic and p-value in your "t-test" and then used a different method in the explicit regression.

I give two examples below where the var.equal = F Welch test has a different p-value than the regression and var.equal = T t-test, which have p-values that do agree.

set.seed(2023)
N <- 5
x <- rep(c(0, 1), N)
e1 <- rnorm(length(x), 0, 1)     # Equal variances
e2 <- rnorm(length(x), 0, x + 1) # Unequal variances
y1 <- x + e1
y2 <- x + e2
summary(lm(y1 ~ x))$coef[2, 4]                      # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = T)$p.val # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = F)$p.val # p = 0.00736509
#
summary(lm(y2 ~ x))$coef[2, 4]                      # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = T)$p.val # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = F)$p.val # p = 0.02251352

Answer 2

While it is true that the regression you apparently run provides as coefficient of the non-constant regressor the difference of the two means, the t-statistic here uses the variance of the OLS estimator, which could be different from the variance used in the statistical two-sample t-test, hence probably the different statistical conclusion.

But without providing the mathematical details of your work, this is all that can be said.

ADDENDUM

This is a response to the post by @Dave. Let two batches of observations, $\{y_i \in \mathbf y_1\}$, and $\{y_i\in \mathbf y_2\}$. Let $\bar y_1$ and $s^2_1$ be the mean and variance of batch 1, and $\bar y_2$ and $s^2_1$ correspondingly for batch 2. Let $n_1$ be the size of batch 1 and $n_2$ be the size of batch 2, and $n=n_1+n_2$. If we put the batches together and consider the regression equation $$y_i = \alpha + \beta\cdot I\{y_i \in \mathbf y_2\}$$,

and we apply Ordinary Least Squares (as the OP did), we will get

$$\hat \alpha = \bar y_1, \quad \hat \beta = \bar y_2 - \bar y_1$$

while the variance of $\hat \beta$ turns out to be

$${\rm Var}(\hat \beta) = \frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}.$$

Note how the sample size of batch 2 divides the variance of batch 1, and the sample size of batch 1 divides the variance of batch 2. So the t-test in the context of OLS regression will use

$$\frac{\hat \beta}{{\rm SD}(\hat \beta)} = \frac{\bar y_2 - \bar y_1}{\left(\frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}\right)^{1/2}}.$$ No assumption is made about equal/unequal sample sizes, and equal/unequal variances.

This is the statistic that must be compared to what stand-alone t-test the OP used.