+ Reply to Thread
Results 1 to 11 of 11

Thread: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

  1. #1
    Points: 474, Level: 9
    Level completed: 48%, Points required for next Level: 26

    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)




    I have a question about ANOVA post-hoc tests in general, with a specific example. In general, I was wondering if you need a significant main effect or interaction to run various post-hoc tests.

    For example, a statistician told me that I did not need a significant main effect to run a Dunnett's test. I have heard conflicting opinions on Bonferroni. One person told me that I cannot use a Bonferroni comparison without a significant interaction. Another person told me that Bonferroni correction eliminates the need for a significant interaction. Does that also mean that you don't need a significant overall effect for a Bonferroni post-hoc?

    Here is my specific example:

    Rats are tested for their ability to recognize an object (familiar or unfamiliar) and receive a drug or vehicle treatment, so I have a 2x2 ANOVA. The unfamilar condition is basically a control condition to account for changes in general activity.

    I have a significant effect of recognition, and a significant effect of drug treatment, but no significant interaction. I did some Bonferroni post-hoc tests to sort out the effect I am really interest in: Vehicle familiar vs drug familiar is significant; vehicle unfamiliar vs drug unfamiliar is not. This is exactly what we expected, but I'm just not sure of legitimacy statistically speaking as far as the post-hocs go.

    Thank you for any advice you can offer.


    Any suggestions?

  2. #2
    Beep
    Points: 61,713, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Discussion EnderPosting AwardCommunity AwardMaster TaggerFrequent Poster
    Dason's Avatar
    Location
    Ames, IA
    Posts
    11,107
    Thanks
    261
    Thanked 2,156 Times in 1,837 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    If those were "pre-planned" (did you know you were going to do these tests even before seeing the results) then I don't think anybody would argue with testing them especially if you control for it using something like Bonferroni.

    One issue is when people do post hoc testing but they only test the differences that look interesting to them or the differences they think will be significant (test the group with the highest mean against the group with the lowest mean). These run into problems because really you hiding the fact that you looked at your data before doing the test and you didn't know before seeing the data if you were going to do this test or not.

  3. #3
    Super Moderator
    Points: 9,766, Level: 66
    Level completed: 29%, Points required for next Level: 284
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,771
    Thanks
    0
    Thanked 140 Times in 126 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Quote Originally Posted by harync View Post

    I have a significant effect of recognition, and a significant effect of drug treatment, but no significant interaction. I did some Bonferroni post-hoc tests to sort out the effect I am really interest in: Vehicle familiar vs drug familiar is significant; vehicle unfamiliar vs drug unfamiliar is not. This is exactly what we expected, but I'm just not sure of legitimacy statistically speaking as far as the post-hocs go.

    Thank you for any advice you can offer.


    Any suggestions?

    In your case there is no need to do any post-hoc analysis. In short, both of your main effects are significant ----you interaction is NOT significant. Thus, all you need to do is look at your main effect F ratios. That's the whole point of the efficiency of a Factorial Design.

  4. #4
    Points: 474, Level: 9
    Level completed: 48%, Points required for next Level: 26

    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Quote Originally Posted by Dragan View Post
    In your case there is no need to do any post-hoc analysis. In short, both of your main effects are significant ----you interaction is NOT significant. Thus, all you need to do is look at your main effect F ratios. That's the whole point of the efficiency of a Factorial Design.
    Thanks for the response.

    The problem is I need to specifically report the significant effect between the "Familiar Vehicle" and "Familiar Drug" groups and lack of a significant effect between the "Unfamiliar Vehicle" and "Unfamiliar Drug" groups. This cannot be determined from the main effect F ratios. As Dason surmised, we planned to do this analysis from the start, so it was a planned comparison.

    I was also interested more broadly in the issue of overall significance and post-hocs. Specifically, if we plan to make certain comparisons (for example Dunnett's test to compare different dose levels to vehicle) we don't need a significant main effect.

  5. #5
    Super Moderator
    Points: 9,766, Level: 66
    Level completed: 29%, Points required for next Level: 284
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,771
    Thanks
    0
    Thanked 140 Times in 126 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Quote Originally Posted by harync View Post
    Thanks for the response.

    The problem is I need to specifically report the significant effect between the "Familiar Vehicle" and "Familiar Drug" groups and lack of a significant effect between the "Unfamiliar Vehicle" and "Unfamiliar Drug" groups. This cannot be determined from the main effect F ratios. As Dason surmised, we planned to do this analysis from the start, so it was a planned comparison.

    I was also interested more broadly in the issue of overall significance and post-hocs. Specifically, if we plan to make certain comparisons (for example Dunnett's test to compare different dose levels to vehicle) we don't need a significant main effect.

    Okee Dokee...I guess where things get confusing to me is when you're using the term "Post-hoc" ---and you're also saying that your experiment "Planned".

  6. #6
    Points: 474, Level: 9
    Level completed: 48%, Points required for next Level: 26

    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Quote Originally Posted by Dragan View Post
    Okee Dokee...I guess where things get confusing to me is when you're using the term "Post-hoc" ---and you're also saying that your experiment "Planned".
    That's where my question about Dunnett's and bonferroni tests come in. If we really are only interested in certain data, such as my example or vehicle vs. dose1, dose2, dose 3, is an ANOVA really necessary? If I know I'm going to make 2 or 3 specific comparisons, what is the most appropriate test to use? Multiple T-tests with some sort of correction?

    The statistics program that I have to use (Graphpad Prism) kind of locks me into the whole ANOVA + post-hoc model so that's why I use that terminology, but I can see how that is confusing.

  7. #7
    Beep
    Points: 61,713, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Discussion EnderPosting AwardCommunity AwardMaster TaggerFrequent Poster
    Dason's Avatar
    Location
    Ames, IA
    Posts
    11,107
    Thanks
    261
    Thanked 2,156 Times in 1,837 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Well with ANOVA you get the added benefit of having every group contributing something - even if they're not in the comparison of interest. This is due to the assumption of equal variance. If that's a valid assumption then doing ANOVA is definitely the route to go because you get added degrees of freedom in your error and just a better estimate of that error variance. If the assumption of equal variance is suspect... well then it might be better to look at other options depending on the situation.

  8. The Following User Says Thank You to Dason For This Useful Post:

    harync (01-10-2012)

  9. #8
    Points: 474, Level: 9
    Level completed: 48%, Points required for next Level: 26

    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Quote Originally Posted by Dason View Post
    Well with ANOVA you get the added benefit of having every group contributing something - even if they're not in the comparison of interest. This is due to the assumption of equal variance. If that's a valid assumption then doing ANOVA is definitely the route to go because you get added degrees of freedom in your error and just a better estimate of that error variance. If the assumption of equal variance is suspect... well then it might be better to look at other options depending on the situation.
    Thanks Dason. That makes a lot of sense.

  10. #9
    R must die
    Points: 24,995, Level: 95
    Level completed: 65%, Points required for next Level: 355
    noetsi's Avatar
    Posts
    4,535
    Thanks
    278
    Thanked 730 Times in 700 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    I am likely missing something here, but I am a bit confused by some of the comments. Planning to do a generic analysis from the beginning (or to look at the omnibus F test to see if some factor was signficant) is not enough to justify planned contrast. This is important because planned contrast have more power than post-hoc tests. You have to have a specific set of means you are comparing determined ahead of time to justify the use of planned contrasts. For example say you had a five level independent variable. You would postulate that level one's mean would be greater (or less than) the mean of the 2 or 3rd level (on the dependent variable). This is done before you run any data at all. If you don't do that you should not used planned contrast analysis.

    Logically you should only look at a post-hoc test of a variable that had a signficant F test. I have never heard that is required that an interaction term must be signficant (to look at main effects with post-hoc tests) that does not make much sense. Nor have I heard that post-hoc methods are not suitable for any main effect that is signficant.

    Tukey's HSD is generally more recommended than Bonferonni.
    This was not what we did in logistic regression. Rather, we transformed the conditional expected value, and made that a linear function of X. This seems odd, because it is odd..

  11. #10
    Points: 474, Level: 9
    Level completed: 48%, Points required for next Level: 26

    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)

    Quote Originally Posted by noetsi View Post
    I am likely missing something here, but I am a bit confused by some of the comments. Planning to do a generic analysis from the beginning (or to look at the omnibus F test to see if some factor was signficant) is not enough to justify planned contrast. This is important because planned contrast have more power than post-hoc tests. You have to have a specific set of means you are comparing determined ahead of time to justify the use of planned contrasts. For example say you had a five level independent variable. You would postulate that level one's mean would be greater (or less than) the mean of the 2 or 3rd level (on the dependent variable). This is done before you run any data at all. If you don't do that you should not used planned contrast analysis.
    In the experiment I described, there are 4 groups and potentially 6 comparisons. Before we ran the experiment, we stated that we were only interested in two specific comparisons. The mean of the familiar-drug group should be lower than the familar-vehicle group. The mean of the unfamiliar-drug group should be the same as the unfamiliar-vehicle group (null hypothesis).

    Due to the limitations of our stats program (Graphpad Prism), we entered the data as a two-way ANOVA and ran bonferroni comparisons. I was asking that given our a priori statement, would it be acceptable to just make two t-test comparisons?

    Quote Originally Posted by noetsi View Post
    Logically you should only look at a post-hoc test of a variable that had a signficant F test. I have never heard that is required that an interaction term must be signficant (to look at main effects with post-hoc tests) that does not make much sense. Nor have I heard that post-hoc methods are not suitable for any main effect that is signficant. Tukey's HSD is generally more recommended than Bonferonni.
    This bring me to a couple more questions. You say that you should only look at a post-hoc test of a variable with a significant F test. I was specifically told by a statistician that one could look at the results of a Dunnett's test without a significant F test. Do you disagree with that? Are there any post-hoc tests that can be run without a significant F test?

    The second question relates to what groups one can look at with one significant main effect and one non-significant main effect. For example, say I am testing vehicle and 4 drug doses in males and females. If there is a significant effect of sex, but not significant effect of drug, would there be any concern with looking at individual doses? It's possible the lack of effect in one of the sexes is hiding a drug effect, but it doesn't show up as an interaction. Conversely, if there is a significant effect of drug, but not sex, would one be forced to combine male and female groups to look at the effect of an individual dose level? That seems problematic to me, especially as a 1 mg/kg dose might work better in females while a 2 mg/kg dose might work better in males. Therefore, would it be appropriate to run post-hocs to compare individual doses within sexes if there is any significant F test? In this circumstance, don't you lose power with Tukey's HSD because you don't want to make every comparison (e.g., male-vehicle vs. female 1 mg/kg, etc)?

    Sorry for the long questions!
    Last edited by harync; 01-11-2012 at 11:39 AM.

  12. #11
    R must die
    Points: 24,995, Level: 95
    Level completed: 65%, Points required for next Level: 355
    noetsi's Avatar
    Posts
    4,535
    Thanks
    278
    Thanked 730 Times in 700 Posts

    Re: Requirements for post-hoc after ANOVA (sig main effect, interaction, etc)


    In the experiment I described, there are 4 groups and potentially 6 comparisons. Before we ran the experiment, we stated that we were only interested in two specific comparisons. The mean of the familiar-drug group should be lower than the familar-vehicle group. The mean of the unfamiliar-drug group should be the same as the unfamiliar-vehicle group (null hypothesis).
    In that case I would calculate a linear (orthoganal) contrast for these and see what you get. This will have more power than any ad hoc test will have.

    This bring me to a couple more questions. You say that you should only look at a post-hoc test of a variable with a significant F test. I was specifically told by a statistician that one could look at the results of a Dunnett's test without a significant F test
    I am not a statistician, I use statistics for analysis. That said this is an issue on which statisticians strongly disagree. Historically it was felt that you should not do follow up tests if the omnibus F test was not signficant. Many now disagree with this, because what the omnibus F test tests is different then what posthoc tests do (including I believe different power as a result). I side with those that believe it makes little logical sense if your omnibus test shows no valid difference to then see if you have valid differences between certain specific means.

    That is a somewhat different question then whether if a F test for a specific variable (a certain main effect) is not significant you would then look at that variable in a post hoc test. But the logic is similar to me - if there is no main effect how can you then look at the means of elements of that main effect (the levels) to see if they differ. That said if a statistician said you can then they are far more knowledable than me. The one thing I would caution, per my statement above, is that there appears to be disagreements among statisticians on related issues (the omnibus F test versus post hoc tests) so you would be wise to look on line and be sure that statisicians actually agree on this point.

    To me it makes little sense to analyze the results of an effect (a variable) that is not signficant. If you think there is a moderator effect occuring that is masking the results of a main effect you should seek to verify that is in fact occuring. There are techniques in regression and (I believe) structural equation models to get at this, although neither are simple. If there is no interaction effect then the level of one of the variables should not influence the impact of another variable on the dependent variable (that is what interaction tests after all). One alternative is to graph the two independent variables impact on the dependent variable at different levels of the other IV and try to determine if the statistics is (for whatever reason) not showing an effect you believe is substantive to exist. Low power, moderation effects etc can all hide important effect sides (meaning that they will show large effect sizes as not statistically signficant).

    Since my knowledge of ANOVA comes from classes and not running it for a living you should take everything I say with a large grain of salt.
    This was not what we did in logistic regression. Rather, we transformed the conditional expected value, and made that a linear function of X. This seems odd, because it is odd..

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats