+ Reply to Thread
Results 1 to 2 of 2

Thread: Correct application of multiple testing methods in projects with several experiments

  1. #1
    Points: 29, Level: 1
    Level completed: 58%, Points required for next Level: 21

    Thanked 0 Times in 0 Posts

    Correct application of multiple testing methods in projects with several experiments


    I have a problem with evaluating how I should correct my alpha and conduct statistical texts when I have several different exeriments in the same project, each with more than two groups to compare.

    - First off, I have a series of measurements that have a control group and three seperate experimental groups that are independent.

    - I have a completely different experiment, with a series of measurements on the same four groups.

    The results of these two experiments are independent of each other.

    So right now, I set up a Holm-Bonferroni process with alpha corrected for all the p-values that were calculated for all the experiments. But I'm not sure if this is actually correct, or if I can get away with doing a seperate correction for either experiment (less stringent corrections).

    Also, right now one-way ANOVAS with Dunett's test were applied to every measurement. However, this also corrects the p-values within that measurement, so in essence I am now correcting my p-values twice, once within the seperate anovas, and once more when I correct my alpha with the H-B process. So if I do the H-B is it than okay to do simple T-tests for every measurement instead of the anovas so the results aren't corrected twice?

    Thank you!

  2. #2
    Points: 1,974, Level: 26
    Level completed: 74%, Points required for next Level: 26

    New Zealand
    Thanked 48 Times in 47 Posts

    Re: Correct application of multiple testing methods in projects with several experime

    Think of Russian roulette with one bullet in a 20 chamber gun. If you use Holm-Bonferroni for just one experiment you limit the risk of shooting yourself in the foot by making a false positive claim, to 1 shot in 20 for that experiment. With two experiments you are taking two shots and doubling the risk. How much risk are you prepared to take of making a false positive claim somewhere in your report? It looks to me that if you want the normal 95% protection against a false positive the should use all the p values as you have set up already. It's just the penalty you pay for having multiple p values.

  3. The Following User Says Thank You to katxt For This Useful Post:

    hlsmith (03-04-2017)

+ Reply to Thread


Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Advertise on Talk Stats