Is SD and SA the population standard deviation or the standard error of the mean? If the previous you need to calculate the standard error ofmthe mean SD/sqrt(100) and then the CI will not overlap.
regards
Hi guys,
I’d like to check the difference of two mean values of two independent samples for significance.
Sample 1: Mean=2.50; SD=0.5; n=100
Sample 2: Mean=2.35; SD=0.5; n=100
With a 95%-probability level the confidence interval is about +/- 0.1 below/ above the mean value, i.e.
confidence interval of sample 1: 2.40 – 2.60
confidence interval of sample 2: 2.25 – 2.45
So the two confidence intervals are overlapping.
If an independent two sample t-test is performed on the same data the t-value is 2.12 and the difference of the mean values is considered significant.
This mystifies me because my understanding was that overlapping confidence intervals indicate that the difference is not significant. So both methods seem to provide contradictory results.
What’s my misconception?
Many thanks in advance.
Last edited by oschl; 04-04-2016 at 10:12 AM.
Is SD and SA the population standard deviation or the standard error of the mean? If the previous you need to calculate the standard error ofmthe mean SD/sqrt(100) and then the CI will not overlap.
regards
This is a common misconception. While it is true that CIs that do not overlap are indeed statistically different, the converse is not true. CIs may overlap by approximately 25% and still have a statistically significant difference. See https://www.cscu.cornell.edu/news/statnews/stnews73.pdf for an explanation.
The confidence intervals you posted are what you get with a 1 Sample t test. Since you're doing a 2 Sample t test, you would want to look at the confidence interval for the difference and see whether that contains zero. The interval for your data is (0.0106, 0.2894), which does not contain zero, therefore your two groups do not appear to be the same. This agrees with the p-value of 0.035 (alpha = 0.05), i.e., you would reject the null hypothesis that the two groups are the same, and conclude that they are different. However, keep in mind that doing the 2-sample t test this way only gives you the Type I error, and it's important to measure the Type II error. That requires calculating the power of the test (1 - beta), and for that you need to come up with a difference between the two groups that you would consider of practical significance, and re-run the t-test using that test difference.
Tweet |