# Thread: Significance testing: Confidence intervals vs. t-test

1. ## Significance testing: Confidence intervals vs. t-test

Hi guys,

I’d like to check the difference of two mean values of two independent samples for significance.
Sample 1: Mean=2.50; SD=0.5; n=100
Sample 2: Mean=2.35; SD=0.5; n=100

With a 95%-probability level the confidence interval is about +/- 0.1 below/ above the mean value, i.e.
confidence interval of sample 1: 2.40 – 2.60
confidence interval of sample 2: 2.25 – 2.45
So the two confidence intervals are overlapping.

If an independent two sample t-test is performed on the same data the t-value is 2.12 and the difference of the mean values is considered significant.

This mystifies me because my understanding was that overlapping confidence intervals indicate that the difference is not significant. So both methods seem to provide contradictory results.
What’s my misconception?

2. ## Re: Significance testing: Confidence intervals vs. t-test

Is SD and SA the population standard deviation or the standard error of the mean? If the previous you need to calculate the standard error ofmthe mean SD/sqrt(100) and then the CI will not overlap.

regards

3. ## Re: Significance testing: Confidence intervals vs. t-test

Originally Posted by oschl
This mystifies me because my understanding was that overlapping confidence intervals indicate that the difference is not significant. So both methods seem to provide contradictory results.
What’s my misconception?
This is a common misconception. While it is true that CIs that do not overlap are indeed statistically different, the converse is not true. CIs may overlap by approximately 25% and still have a statistically significant difference. See https://www.cscu.cornell.edu/news/statnews/stnews73.pdf for an explanation.

4. ## Re: Significance testing: Confidence intervals vs. t-test

Originally Posted by Miner
This is a common misconception. While it is true that CIs that do not overlap are indeed statistically different, the converse is not true. CIs may overlap by approximately 25% and still have a statistically significant difference. See https://www.cscu.cornell.edu/news/statnews/stnews73.pdf for an explanation.
Thank you very much. That clarifies it!

5. ## Re: Significance testing: Confidence intervals vs. t-test

The confidence intervals you posted are what you get with a 1 Sample t test. Since you're doing a 2 Sample t test, you would want to look at the confidence interval for the difference and see whether that contains zero. The interval for your data is (0.0106, 0.2894), which does not contain zero, therefore your two groups do not appear to be the same. This agrees with the p-value of 0.035 (alpha = 0.05), i.e., you would reject the null hypothesis that the two groups are the same, and conclude that they are different. However, keep in mind that doing the 2-sample t test this way only gives you the Type I error, and it's important to measure the Type II error. That requires calculating the power of the test (1 - beta), and for that you need to come up with a difference between the two groups that you would consider of practical significance, and re-run the t-test using that test difference.

 Tweet

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts