0
$\begingroup$

I have two batches of data and I want to check whether the response variable is different across the two batches. I first did a t-test and found that the means of the response variables in the two batches differ significantly. I then run a regression with an intercept and a Group 2 dummy. The coefficient of the Group 2 dummy is insignificant. This regression result seems to be inconsistent with the t-test result. Could anyone please explain this happens and provide a simple example? Thanks.

$\endgroup$
3
  • 2
    $\begingroup$ Hi: This won't answer your question but what did you use for the denominator of your t-test ? By this I mean, how did you calculate the sample variance of the difference estimate Once that expression is known, it could possibly explain some of the the difference Actually, now that I think about it, my advice to you write out exactly what you did in both cases notationally by using equations in latex. ( or whatever it is called here. MathJax ? ) It can kind of be roughly guessed by a reader but it's easier to see it in lights :). $\endgroup$
    – mark leeds
    Commented Nov 17, 2023 at 8:03
  • 2
    $\begingroup$ Adding to @mark leeds suggestionsss: could you add how you computed the regression (e.g. Python or R syntax and what the dataframe looks like (a table of headers and a few entries). Otherwise it's mainly guessing what you may or may not have done. $\endgroup$
    – AKdemy
    Commented Nov 17, 2023 at 11:20
  • $\begingroup$ Thanks for your answers. In fact, I used ChatGPT to do this so I don't know the math models behind these. I will dig deeper into the models. $\endgroup$
    – Justin
    Commented Nov 19, 2023 at 16:31

2 Answers 2

1
$\begingroup$

There are multiple ways to perform various test that can be defended, depending on the context, but they are not, necessarily, equivalent.

The typical way to calculate the standard errors in an OLS linear model assumes equal variances in the two groups (constant-variance error term for the general model). This is the default calculation in most software, and then the confidence intervals and p-values are derived from that standard error.

There is a variant of the t-test called Welch's t-test that assumes (or at least allows for) unequal variances in to two groups. It sounds like you might have used this calculation for what you are calling the "t-test". For example, this is the default calculation in the t.test function in R software.

While the usual t-test will agree with the usual OLS linear model coefficient test, Welch's test does not agree with the usual t-test.

It sounds like you used some variant of the usual t-test, such as Welch's test, to calculate the standard error and resulting t-statistic and p-value in your "t-test" and then used a different method in the explicit regression.

I give two examples below where the var.equal = F Welch test has a different p-value than the regression and var.equal = T t-test, which have p-values that do agree.

set.seed(2023)
N <- 5
x <- rep(c(0, 1), N)
e1 <- rnorm(length(x), 0, 1)     # Equal variances
e2 <- rnorm(length(x), 0, x + 1) # Unequal variances
y1 <- x + e1
y2 <- x + e2
summary(lm(y1 ~ x))$coef[2, 4]                      # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = T)$p.val # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = F)$p.val # p = 0.00736509
#
summary(lm(y2 ~ x))$coef[2, 4]                      # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = T)$p.val # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = F)$p.val # p = 0.02251352
$\endgroup$
1
  • $\begingroup$ What you so nicely explained is EXACTLY why I asked him to show his work and specifically show what he was doing. But it's been a while now so maybe he worked it out ? $\endgroup$
    – mark leeds
    Commented Nov 20, 2023 at 19:11
1
$\begingroup$

While it is true that the regression you apparently run provides as coefficient of the non-constant regressor the difference of the two means, the t-statistic here uses the variance of the OLS estimator, which could be different from the variance used in the statistical two-sample t-test, hence probably the different statistical conclusion.

But without providing the mathematical details of your work, this is all that can be said.

ADDENDUM

This is a response to the post by @Dave. Let two batches of observations, $\{y_i \in \mathbf y_1\}$, and $\{y_i\in \mathbf y_2\}$. Let $\bar y_1$ and $s^2_1$ be the mean and variance of batch 1, and $\bar y_2$ and $s^2_1$ correspondingly for batch 2. Let $n_1$ be the size of batch 1 and $n_2$ be the size of batch 2, and $n=n_1+n_2$. If we put the batches together and consider the regression equation $$y_i = \alpha + \beta\cdot I\{y_i \in \mathbf y_2\}$$,

and we apply Ordinary Least Squares (as the OP did), we will get

$$\hat \alpha = \bar y_1, \quad \hat \beta = \bar y_2 - \bar y_1$$

while the variance of $\hat \beta$ turns out to be

$${\rm Var}(\hat \beta) = \frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}.$$

Note how the sample size of batch 2 divides the variance of batch 1, and the sample size of batch 1 divides the variance of batch 2. So the t-test in the context of OLS regression will use

$$\frac{\hat \beta}{{\rm SD}(\hat \beta)} = \frac{\bar y_2 - \bar y_1}{\left(\frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}\right)^{1/2}}.$$ No assumption is made about equal/unequal sample sizes, and equal/unequal variances.

This is the statistic that must be compared to what stand-alone t-test the OP used.

$\endgroup$
4
  • $\begingroup$ Equal-variance vs unequal-variance standard error, you mean? $\endgroup$
    – Dave
    Commented Nov 20, 2023 at 4:15
  • $\begingroup$ @Dave ...compared also to the standard error of a regression coefficient. $\endgroup$ Commented Nov 20, 2023 at 15:25
  • $\begingroup$ The usual standard error for an OLS coefficient coincides with the standard error in an equal-variance two-sample t-test. Not all software uses the equal-variance test by default (t.test in R uses Welch's test by default, for example), so I could believe that the difference is because the regression calculation used an equal-variance assumption while the standard t-test used an unequal-variance calculation. $\endgroup$
    – Dave
    Commented Nov 20, 2023 at 15:31
  • $\begingroup$ @Dave This is why everybody is telling the OP to post their math, $\endgroup$ Commented Nov 20, 2023 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.