# t-test result inconsistent with regression

I have two batches of data and I want to check whether the response variable is different across the two batches. I first did a t-test and found that the means of the response variables in the two batches differ significantly. I then run a regression with an intercept and a Group 2 dummy. The coefficient of the Group 2 dummy is insignificant. This regression result seems to be inconsistent with the t-test result. Could anyone please explain this happens and provide a simple example? Thanks.

• Hi: This won't answer your question but what did you use for the denominator of your t-test ? By this I mean, how did you calculate the sample variance of the difference estimate Once that expression is known, it could possibly explain some of the the difference Actually, now that I think about it, my advice to you write out exactly what you did in both cases notationally by using equations in latex. ( or whatever it is called here. MathJax ? ) It can kind of be roughly guessed by a reader but it's easier to see it in lights :). Nov 17 at 8:03
• Adding to @mark leeds suggestionsss: could you add how you computed the regression (e.g. Python or R syntax and what the dataframe looks like (a table of headers and a few entries). Otherwise it's mainly guessing what you may or may not have done. Nov 17 at 11:20
• Thanks for your answers. In fact, I used ChatGPT to do this so I don't know the math models behind these. I will dig deeper into the models. Nov 19 at 16:31

There are multiple ways to perform various test that can be defended, depending on the context, but they are not, necessarily, equivalent.

The typical way to calculate the standard errors in an OLS linear model assumes equal variances in the two groups (constant-variance error term for the general model). This is the default calculation in most software, and then the confidence intervals and p-values are derived from that standard error.

There is a variant of the t-test called Welch's t-test that assumes (or at least allows for) unequal variances in to two groups. It sounds like you might have used this calculation for what you are calling the "t-test". For example, this is the default calculation in the t.test function in R software.

While the usual t-test will agree with the usual OLS linear model coefficient test, Welch's test does not agree with the usual t-test.

It sounds like you used some variant of the usual t-test, such as Welch's test, to calculate the standard error and resulting t-statistic and p-value in your "t-test" and then used a different method in the explicit regression.

I give two examples below where the var.equal = F Welch test has a different p-value than the regression and var.equal = T t-test, which have p-values that do agree.

set.seed(2023)
N <- 5
x <- rep(c(0, 1), N)
e1 <- rnorm(length(x), 0, 1)     # Equal variances
e2 <- rnorm(length(x), 0, x + 1) # Unequal variances
y1 <- x + e1
y2 <- x + e2
summary(lm(y1 ~ x))$coef[2, 4] # p = 0.006424082 t.test(y1[x == 0], y1[x == 1], var.equal = T)$p.val # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = F)$p.val # p = 0.00736509 # summary(lm(y2 ~ x))$coef[2, 4]                      # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = T)$p.val # p = 0.01837531 t.test(y2[x == 0], y2[x == 1], var.equal = F)$p.val # p = 0.02251352

• What you so nicely explained is EXACTLY why I asked him to show his work and specifically show what he was doing. But it's been a while now so maybe he worked it out ? Nov 20 at 19:11

While it is true that the regression you apparently run provides as coefficient of the non-constant regressor the difference of the two means, the t-statistic here uses the variance of the OLS estimator, which could be different from the variance used in the statistical two-sample t-test, hence probably the different statistical conclusion.

But without providing the mathematical details of your work, this is all that can be said.

This is a response to the post by @Dave. Let two batches of observations, $$\{y_i \in \mathbf y_1\}$$, and $$\{y_i\in \mathbf y_2\}$$. Let $$\bar y_1$$ and $$s^2_1$$ be the mean and variance of batch 1, and $$\bar y_2$$ and $$s^2_1$$ correspondingly for batch 2. Let $$n_1$$ be the size of batch 1 and $$n_2$$ be the size of batch 2, and $$n=n_1+n_2$$. If we put the batches together and consider the regression equation $$y_i = \alpha + \beta\cdot I\{y_i \in \mathbf y_2\}$$,

and we apply Ordinary Least Squares (as the OP did), we will get

$$\hat \alpha = \bar y_1, \quad \hat \beta = \bar y_2 - \bar y_1$$

while the variance of $$\hat \beta$$ turns out to be

$${\rm Var}(\hat \beta) = \frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}.$$

Note how the sample size of batch 2 divides the variance of batch 1, and the sample size of batch 1 divides the variance of batch 2. So the t-test in the context of OLS regression will use

$$\frac{\hat \beta}{{\rm SD}(\hat \beta)} = \frac{\bar y_2 - \bar y_1}{\left(\frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}\right)^{1/2}}.$$ No assumption is made about equal/unequal sample sizes, and equal/unequal variances.

This is the statistic that must be compared to what stand-alone t-test the OP used.

• Equal-variance vs unequal-variance standard error, you mean?
– Dave
Nov 20 at 4:15
• @Dave ...compared also to the standard error of a regression coefficient. Nov 20 at 15:25
• The usual standard error for an OLS coefficient coincides with the standard error in an equal-variance two-sample t-test. Not all software uses the equal-variance test by default (t.test in R uses Welch's test by default, for example), so I could believe that the difference is because the regression calculation used an equal-variance assumption while the standard t-test used an unequal-variance calculation.
– Dave
Nov 20 at 15:31
• @Dave This is why everybody is telling the OP to post their math, Nov 20 at 15:39