Bonferroni correction

[poons]

Hello,

I'm struggling with applying the Bonferroni correction in my analysis, more specifically figuring out the number of comparisons I have to control for.

I have several statements which participants (divided into three independent groups) rated on a ordinal scale. I'm interested if there are differences for each statement between the groups. For that purpose, I performed a Kruskal-Wallis H test with Dunn's post-hoc test and a corrected alpha level of 0.05/3 for each statement (as I do not draw comparisons between statements). Now I'm wondering if the correction should rather take all pair-wise comparisons across all statements (0.05 / (nr of statements x 3)) into account in order to control for false-positives across the whole analysis?

EDIT: After thinking about it again, would it also make sense to control for the multiple Kruskal-Wallis H test in the first place? If so, how could that be accomplished?

Thank you in advance!

 

[fed2]

im about 92.75% confident that it is correct the way you have it. But, since this is actually two questions, there may be some alpha inflation there!

 

[katxt]

One school of thought says you should decide before you analyze just how many comparisons you want to make and so how many p values you are going to generate from those comparisons. Use that number for your Bonferroni divider, and resist the temptation to look for extras after.

 

[fed2]

yes but its not necessary that all of the alpha control be by bonferonni. this is more in keeping with the fda 'gatekeeping' or 'graphical' approach. It is conceptually alot simpler if you only use one procedure for sure, but it is not necessary. It looks accurate to me but i could be convinced otherwise.

 

[poons]

Thanks both for your input so far.

@fed2 Just to be sure, you mean doing it individually for each post-hoc (0.05/3) is the correct way or across all of them?

@katxt Thanks. My question was more about how do I go about it in the first place, i.e. how do I decide before? Take all comparisions into account or just for each comparison (as I'm not comparing across statements)?

 

[fed2]

yes 0.05/3. I think its probably overkill to break up the alpha over three endpoints like this for research purposes, but may be required in some settings.

 

[katxt]

It depends on how much risk you are prepared to take of making a false claim that there is a difference, and that depends on the negative consequences of making such a claim.