+ Reply to Thread
Results 1 to 8 of 8

Thread: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

  1. #1
    Points: 30, Level: 1
    Level completed: 60%, Points required for next Level: 20

    Posts
    3
    Thanks
    5
    Thanked 0 Times in 0 Posts

    Question Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.




    I planned to analyse some data with several ANOVAs (I have many different dependent variables), but much of the data violates the ANOVA assumptions, so they require Welch's ANOVA or the non-parametric Kruskal-Wallis test instead.

    Using SPSS, I generated effect sizes (partial eta squared) and observed power values for the data which I analysed using an ANOVA.

    I calculated estimated omega squared as an effect size for the Welch's ANOVAs using this formula:
    est. omega squared = DFbetweensjcts(F-1) / (DFbetweensjcts(F-1)+N)

    I calculated eta squared as an effect size for the Kruskal-Wallis test using this formula:
    eta squared = (Kruskal-Wallis chi-square value) / (N-1)

    The problem is, I am unsure how to calculate observed power for the Welch's ANOVA or Kruskal-Wallis test. Is it possible to do this? If so, how?

    Also, I found the formulae to calculate omega squared and eta squared somewhere online, so I'm a little unsure whether these are correct.

    Any help would be really appreciated!

  2. #2
    Points: 1,741, Level: 24
    Level completed: 41%, Points required for next Level: 59

    Posts
    230
    Thanks
    37
    Thanked 68 Times in 59 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

    Quote Originally Posted by ble_21 View Post
    I planned to analyse some data with several ANOVAs (I have many different dependent variables), but much of the data violates the ANOVA assumptions, so they require Welch's ANOVA or the non-parametric Kruskal-Wallis test instead.

    Using SPSS, I generated effect sizes (partial eta squared) and observed power values for the data which I analysed using an ANOVA.

    I calculated estimated omega squared as an effect size for the Welch's ANOVAs using this formula:
    est. omega squared = DFbetweensjcts(F-1) / (DFbetweensjcts(F-1)+N)

    I calculated eta squared as an effect size for the Kruskal-Wallis test using this formula:
    eta squared = (Kruskal-Wallis chi-square value) / (N-1)

    The problem is, I am unsure how to calculate observed power for the Welch's ANOVA or Kruskal-Wallis test. Is it possible to do this? If so, how?

    Also, I found the formulae to calculate omega squared and eta squared somewhere online, so I'm a little unsure whether these are correct.

    Any help would be really appreciated!
    What do you hope to accomplish by conducting a post-hoc power analysis? What information will the "observed" power analysis provide for you?

  3. The Following User Says Thank You to ondansetron For This Useful Post:

    ble_21 (08-18-2017)

  4. #3
    Points: 30, Level: 1
    Level completed: 60%, Points required for next Level: 20

    Posts
    3
    Thanks
    5
    Thanked 0 Times in 0 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

    I would like to calculate power so that I can report whether any non-significant results were under-powered, because if so, then I will know that further research is needed with larger sample sizes. Also, some of the journals I hope to submit my results to ask that power is reported so that it can inform the design of future research.

  5. #4
    Points: 1,741, Level: 24
    Level completed: 41%, Points required for next Level: 59

    Posts
    230
    Thanks
    37
    Thanked 68 Times in 59 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

    Quote Originally Posted by ble_21 View Post
    I would like to calculate power so that I can report whether any non-significant results were under-powered, because if so, then I will know that further research is needed with larger sample sizes. Also, some of the journals I hope to submit my results to ask that power is reported so that it can inform the design of future research.
    I see. Well, the appropriate way to answer your question is an a priori power analysis, meaning that it is done before the data are collected and seen. You specify a set of assumptions (alpha, effect size, etc) and a desired power to yield a sample size estimate to ensure the specified power level according to your assumptions. If you don't meet the sample size requirement and fail to reach significance on a test, you could propose underpowering as a reason.

    However, post-hoc power analysis is of little value to answering this question, I believe. In a non-technical way, seeing the data and basing the power calculations off of it is like seeing the answers to a test and concluding you really earned your grade on the exam.

    Maybe Dason could weigh in to verify or add theoretical clarification. As far as I know, though, post-hoc power analysis isn't entirely valid for assessing if your study was underpowered. I would probably prefer to state power as a potential reason and say a new study should include an a priori power analysis to assess the reproducibility of the conclusions.

  6. The Following User Says Thank You to ondansetron For This Useful Post:

    ble_21 (08-18-2017)

  7. #5
    Human
    Points: 12,676, Level: 73
    Level completed: 57%, Points required for next Level: 174
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,362
    Thanks
    455
    Thanked 462 Times in 402 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

    I agree with what Ondansetron said.

    Look at this and this and search for "post hoc powere of test".

  8. The Following 2 Users Say Thank You to GretaGarbo For This Useful Post:

    ble_21 (08-18-2017), ondansetron (08-17-2017)

  9. #6
    Points: 1,741, Level: 24
    Level completed: 41%, Points required for next Level: 59

    Posts
    230
    Thanks
    37
    Thanked 68 Times in 59 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

    Little did I know, I already have The Abuse of Power article downloaded and in a reading list. For the OP, read the article by Hoenig for further background, and use it as a reference to show the reviewers that it's not a well advised plan to utilize a post-hoc power analysis-- that is, assuming they asked for post-hoc and not a priori.

  10. The Following User Says Thank You to ondansetron For This Useful Post:

    ble_21 (08-18-2017)

  11. #7
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

    I agree with ondansetron and GretaGarbo.

    To expand on their comments a little, "post-hoc power" (i.e., doing a power analysis where you just plug in the observed effect size, observed sample size, etc.) produces a "power" estimate that is simply a transformation of the p-value that you already observed. A p-value of p = .05 translates to post-hoc power = 50%. If you rejected the null, then post-hoc power will be >50%. If you failed to reject the null, then post-hoc power will be <50%.*

    What this means is that if you already know the p-value for your analysis, then post-hoc power literally adds no new information. It is mathematically impossible to have a non-significant p-value & high post-hoc power; and it is impossible to have a significant p-value and low post-hoc power.

    (* Note that this is slightly oversimplified; in truth the post-hoc power value that corresponds to observed p = .05 is 50% for large-sample z tests; for finite samples, this equilibrium power value approaches 50% from above as the sample size grows, but is slightly greater than 50% for small samples. See Russ Lenth's excellent paper on the topic.)

    Quote Originally Posted by ble_21 View Post
    Also, some of the journals I hope to submit my results to ask that power is reported so that it can inform the design of future research.
    Yes, but what they want is a prospective power analysis, not a post-hoc power analysis. Since you can't do a truly prospective power analysis (since you can't go back in time), what you can do instead is to present a power analysis for a range of small, medium, and large effect sizes (NOT just for your observed effect size) for your observed sample size, so that readers have an idea of what your a priori chances of detecting common effect sizes were. That would actually be informative, unlike post-hoc power.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  12. The Following 3 Users Say Thank You to Jake For This Useful Post:

    ble_21 (08-18-2017), ondansetron (08-17-2017), rogojel (08-18-2017)

  13. #8
    Points: 30, Level: 1
    Level completed: 60%, Points required for next Level: 20

    Posts
    3
    Thanks
    5
    Thanked 0 Times in 0 Posts

    Re: Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.


    Thanks for your comments, everyone. These are all really useful things to know - I didn't realize that observed power is just a transformation of the p-value, but this seems pretty obvious in hindsight, after looking at my data!

    I'll take on your advice and present a power analysis for a range of effect sizes for my observed sample size.

    Thanks!

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats