Correct application of multiple testing methods in projects with several experiments

#1
Hi!

I have a problem with evaluating how I should correct my alpha and conduct statistical texts when I have several different exeriments in the same project, each with more than two groups to compare.

- First off, I have a series of measurements that have a control group and three seperate experimental groups that are independent.

- I have a completely different experiment, with a series of measurements on the same four groups.

The results of these two experiments are independent of each other.

So right now, I set up a Holm-Bonferroni process with alpha corrected for all the p-values that were calculated for all the experiments. But I'm not sure if this is actually correct, or if I can get away with doing a seperate correction for either experiment (less stringent corrections).

Also, right now one-way ANOVAS with Dunett's test were applied to every measurement. However, this also corrects the p-values within that measurement, so in essence I am now correcting my p-values twice, once within the seperate anovas, and once more when I correct my alpha with the H-B process. So if I do the H-B is it than okay to do simple T-tests for every measurement instead of the anovas so the results aren't corrected twice?

Thank you!
 

katxt

Well-Known Member
#2
Re: Correct application of multiple testing methods in projects with several experime

Think of Russian roulette with one bullet in a 20 chamber gun. If you use Holm-Bonferroni for just one experiment you limit the risk of shooting yourself in the foot by making a false positive claim, to 1 shot in 20 for that experiment. With two experiments you are taking two shots and doubling the risk. How much risk are you prepared to take of making a false positive claim somewhere in your report? It looks to me that if you want the normal 95% protection against a false positive the should use all the p values as you have set up already. It's just the penalty you pay for having multiple p values.