What is a two sample unequal variance t test
It is clear that unequal variances do change how the test performs. Is unfair to say that the test has "detected" unequal variances, but Have "detected" that means are unequal, because they aren't. Made a difference in how the pooled t test works. Obviously, the change from equal variances to unequal variances has The 'null distribution' (distribution when $H_0$ is true) has changed substantially. Now the test is falsely rejecting about 30% of the time-much more than 5% of the time. Now let's see what happens if we keep everything exactly the same-except that we change the population variances to be unequal, with $\sigma_1^2 = 16$ and $\sigma_2^2 = 1.$ set.seed(818) Just 'as advertised': The pooled 2-sample t test has incorrectly rejected $H_0$ in almost exactly 5% of the tests on one million sets of two samples from
Results of a million such pooled 2-sample t tests. We could discuss the theory to show that this
However, 5% of the time, a pooled test at the 5% level will makeĪ mistake, rejecting $H_0$ with a P-value $ < 0.05.$ From the simulation, we know that $\mu_1 - \mu_2 = 50.$ (Also that $\sigma_1^2 = \sigma_2^2 = 1.)$Īnd the test has (correctly) failed to reject $H_0.$ True difference in means is not equal to 0Īll is well. We reject $H_0$ atĬomparing two specific such samples, what outputĭo we get from the pooled 2-sample t test? set.seed(1234) Let's consider a sample of size $n_1 = 10$ from You're talking about a pooled 2-sample t test, of