Adjustment for multiple comparisons

#1
Hi there I am hoping you can help me out with a confusing topic.

I have a 2x3x4 repeated measured anova. I have a significant 3-way interaction, and I want to make sure that I am using the correct post hoc comparisons and not violating any key statistics theory.

I have run the statistics in SPSS and have adjusted for multiple comparisons using SIDAK but I want to make sure I understand how many adjustments are being made and if the p values are being corrected appropriately.

My post hoc tests have analysed the following:

AxB at each level of C

AxC at each level of B

BXC at each level of A

I am just trying to determine what correction factor is appropriate. For example b x c at each level of A compares 2 means 12 times. Here I am assuming that I am not adjusting the p value because it is a 2 means, despite the fact that I am comparing the 2 means 12 time at different combinations of B and C.

Similarly for a x c at each level of B I am comparing 3 means (as b has 3 levels) at each combination of a x c (8 combinations). Here I am assuming that I would I be dividing the p value by 3 and not by the combinations axc (8 or by the means multiplies by the # combinations 8x3=24).

Similarly, for axb at each level of c I am comparing 4 means (5 comparisons total) at each combination of a x b (6 total).Here I am assuming that I would be dividing the p value by 5 and not by the combinations x the different means hence 30.

I am also assuming that I would not be dividing the p value by the sum of all the above combinations, does that sound correct?
 
#3
Thanks for the response, I have used a SIDAK correction with SPSS which adjusts

AxB at each level of C by 5 as there are 4 levels of C

AxC at each level of B by 3 as there are 3 levels of B

BXC at each level of A by 0 as there are only 2 levels

I wanted to make sure that this was correct. I understand that the Holm-SIdak may be more robust, but wonder if you have any insight into is the appropriate corrections have been made in SPSS
 
#4
To Follow up I just received advice to divide alpha by 9 (factor A levels +factor B levels + factor c levels) however, SPSS in it's post-hoc adjustment simply divides in a way that I have suggested above. Do you have any suggestions based on this?
Thanks
 

hlsmith

Less is more. Stay pure. Stay poor.
#5
General information related to familywise error rate:

Sidak adjusts in order to hold the level of significance at 0.05 (or which ever level you select), considering you are making multiple tests.

The method you mention in post #4 is the Bonferroni correction, one of the more commonly used due to ease in understanding. It just divides the alpha by the number of comparisons or multiplies the p-values by the number of comparisons. The number of tests (comparisons) equals the total number of possible pairwise comparisons. Since the false discovery rate is asymptotic and does result in a straightline to correct with when you have more tests, the Bonferroni unlike the Sidak will provide a much lower cutoff for significance when you have more tests (I believe), eventually lower than your initial cutoff level you may have wanted to maintain after correcting for false discovery.