+ Reply to Thread
Results 1 to 5 of 5

Thread: Significance testing: Confidence intervals vs. t-test

  1. #1
    Points: 131, Level: 2
    Level completed: 62%, Points required for next Level: 19

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Significance testing: Confidence intervals vs. t-test




    Hi guys,

    I’d like to check the difference of two mean values of two independent samples for significance.
    Sample 1: Mean=2.50; SD=0.5; n=100
    Sample 2: Mean=2.35; SD=0.5; n=100

    With a 95%-probability level the confidence interval is about +/- 0.1 below/ above the mean value, i.e.
    confidence interval of sample 1: 2.40 – 2.60
    confidence interval of sample 2: 2.25 – 2.45
    So the two confidence intervals are overlapping.

    If an independent two sample t-test is performed on the same data the t-value is 2.12 and the difference of the mean values is considered significant.

    This mystifies me because my understanding was that overlapping confidence intervals indicate that the difference is not significant. So both methods seem to provide contradictory results.
    What’s my misconception?

    Many thanks in advance.
    Last edited by oschl; 04-04-2016 at 10:12 AM.

  2. #2
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Significance testing: Confidence intervals vs. t-test

    Is SD and SA the population standard deviation or the standard error of the mean? If the previous you need to calculate the standard error ofmthe mean SD/sqrt(100) and then the CI will not overlap.

    regards

  3. #3
    TS Contributor
    Points: 14,811, Level: 78
    Level completed: 91%, Points required for next Level: 39
    Miner's Avatar
    Location
    Greater Milwaukee area
    Posts
    1,171
    Thanks
    34
    Thanked 405 Times in 363 Posts

    Re: Significance testing: Confidence intervals vs. t-test

    Quote Originally Posted by oschl View Post
    This mystifies me because my understanding was that overlapping confidence intervals indicate that the difference is not significant. So both methods seem to provide contradictory results.
    What’s my misconception?
    This is a common misconception. While it is true that CIs that do not overlap are indeed statistically different, the converse is not true. CIs may overlap by approximately 25% and still have a statistically significant difference. See https://www.cscu.cornell.edu/news/statnews/stnews73.pdf for an explanation.

  4. #4
    Points: 131, Level: 2
    Level completed: 62%, Points required for next Level: 19

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Significance testing: Confidence intervals vs. t-test

    Quote Originally Posted by Miner View Post
    This is a common misconception. While it is true that CIs that do not overlap are indeed statistically different, the converse is not true. CIs may overlap by approximately 25% and still have a statistically significant difference. See https://www.cscu.cornell.edu/news/statnews/stnews73.pdf for an explanation.
    Thank you very much. That clarifies it!

  5. #5
    Points: 23, Level: 1
    Level completed: 45%, Points required for next Level: 27

    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Significance testing: Confidence intervals vs. t-test


    The confidence intervals you posted are what you get with a 1 Sample t test. Since you're doing a 2 Sample t test, you would want to look at the confidence interval for the difference and see whether that contains zero. The interval for your data is (0.0106, 0.2894), which does not contain zero, therefore your two groups do not appear to be the same. This agrees with the p-value of 0.035 (alpha = 0.05), i.e., you would reject the null hypothesis that the two groups are the same, and conclude that they are different. However, keep in mind that doing the 2-sample t test this way only gives you the Type I error, and it's important to measure the Type II error. That requires calculating the power of the test (1 - beta), and for that you need to come up with a difference between the two groups that you would consider of practical significance, and re-run the t-test using that test difference.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats