+ Reply to Thread
Results 1 to 11 of 11

Thread: What if ANOVA post-hoc isn't significant, but T-test is?

  1. #1
    Points: 2,784, Level: 32
    Level completed: 23%, Points required for next Level: 116

    Posts
    10
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Unhappy What if ANOVA post-hoc isn't significant, but T-test is?




    The general question I have is, what do I do when a post-hoc difference comparison of 2 means form an ANOVA (that has a total of 4 groups) shows no statistical difference, but looking at the raw data, the means of those groups, the stdev of those groups, and running a t-test on them indicates they are different?

    The details are as follows: I recently ran an ANOVA looking at the effect of treatment or placebo on group A and group B mean values of a hormone. The ANOVA post-hocs indicated significant differences in treated vs placebo for group B, but not group A. But Group A shows the bigger difference in means from treatment versus placebo. Also, Group A (treated or placebo) show tighter stdev than the Group B values. Finally, separate t-tests for Groups A and B indicate significant treatment effects.

    I'm not sure which statistics to trust here. I originally went with the ANOVA (using Statistica), but the raw data and mean values just seem too different for Group A and the t-test result seems more believable based on this.

    Thoughts?

  2. #2
    Beep
    Points: 63,393, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Discussion EnderPosting AwardCommunity AwardFrequent PosterActivity Award
    Dason's Avatar
    Location
    Ames, IA
    Posts
    11,333
    Thanks
    266
    Thanked 2,202 Times in 1,881 Posts
    Do you have different sample sizes for group A and B? Are the standard deviations similar for all groups?

  3. #3
    Points: 2,784, Level: 32
    Level completed: 23%, Points required for next Level: 116

    Posts
    10
    Thanks
    1
    Thanked 0 Times in 0 Posts
    A: Groups are n=15 (placebo) and n=11 (treat). Stdev are 9 and 10
    B: Groups are n=15 (placebo) and n=11 (treat). Stdev are 33 and 57

    Group A means are 57 and 29
    Group B means are 289 and 233

    Group A are genetically different than Group B so we expect them to be lower regardless of treatment and they clearly are. ANOVA confirmed this as well.

  4. #4
    Beep
    Points: 63,393, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Discussion EnderPosting AwardCommunity AwardFrequent PosterActivity Award
    Dason's Avatar
    Location
    Ames, IA
    Posts
    11,333
    Thanks
    266
    Thanked 2,202 Times in 1,881 Posts
    Do you know the assumptions that Anova makes?

    http://en.wikipedia.org/wiki/Analysi...tions_of_ANOVA

    Anova is assuming all groups have an equal variance.

  5. #5
    Points: 2,784, Level: 32
    Level completed: 23%, Points required for next Level: 116

    Posts
    10
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Right, but this assumption is less important when sample sizes are similar. Do you you think 11 and 15 are not similar enough?

  6. #6
    Ninja say what!?!
    Points: 8,297, Level: 61
    Level completed: 49%, Points required for next Level: 153
    Link's Avatar
    Posts
    1,165
    Thanks
    37
    Thanked 84 Times in 76 Posts
    How exactly are you running your ANOVA? Are you running an analysis on group A and an independent one on group B?

  7. #7
    Ninja say what!?!
    Points: 8,297, Level: 61
    Level completed: 49%, Points required for next Level: 153
    Link's Avatar
    Posts
    1,165
    Thanks
    37
    Thanked 84 Times in 76 Posts
    Quote Originally Posted by hwc007 View Post
    Right, but this assumption is less important when sample sizes are similar. Do you you think 11 and 15 are not similar enough?
    11 and 15 are the sample sizes. I think he was referring to the stdev's

  8. #8
    Points: 2,784, Level: 32
    Level completed: 23%, Points required for next Level: 116

    Posts
    10
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Link View Post
    11 and 15 are the sample sizes. I think he was referring to the stdev's
    Yes, but the assumption of equal variances is less important when you have 1) similar sample sizes or 2) large sample sizes. I don't have the latter, but I do have the former...though it could be argued that 11 and 15 are not similar enough.

    I am running one anova. Group A placebo, Group A treat, Group B placebo, Group B treat. 4 groups total. Group A placebo and treat are the ones that aren't coming out different and yet they have the tighter data.

  9. #9
    Ninja say what!?!
    Points: 8,297, Level: 61
    Level completed: 49%, Points required for next Level: 153
    Link's Avatar
    Posts
    1,165
    Thanks
    37
    Thanked 84 Times in 76 Posts
    If you're doing one analysis, then you're violating that assumption. I don't think it matters that you have similar sample sizes. The std dev's in groups A and B are too different. My impression is that the t-test is the way to go.

  10. #10
    Points: 2,784, Level: 32
    Level completed: 23%, Points required for next Level: 116

    Posts
    10
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Thanks guys. I just log transformed the data and ran the ANOVA and it showed the differences I was expecting (and confirmed the t-test results too).

  11. #11
    Beep
    Points: 63,393, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Discussion EnderPosting AwardCommunity AwardFrequent PosterActivity Award
    Dason's Avatar
    Location
    Ames, IA
    Posts
    11,333
    Thanks
    266
    Thanked 2,202 Times in 1,881 Posts

    I was going to recommend a log transform. Were you doing a Welch t-test before?

+ Reply to Thread

           




Similar Threads

  1. Replies: 0
    Last Post: 10-11-2010, 03:53 PM
  2. Replies: 3
    Last Post: 06-30-2010, 03:09 PM
  3. Non-significant Omnibus F with significant Post-hoc
    By gravitykiller in forum Statistics
    Replies: 9
    Last Post: 06-15-2010, 04:32 PM
  4. Significant ANOVA but insignificant post hocs
    By psychgrl778 in forum Statistics
    Replies: 2
    Last Post: 01-20-2010, 08:15 PM
  5. Post hoc on ANOVA significant interactions
    By JenS in forum Statistics
    Replies: 1
    Last Post: 07-27-2007, 08:00 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats