Comparing distributions, Two-Sample KS-test and Bonferroni

Hi there,

The following is my problem for which I’m grateful to have any advice/opinions:
I measured the distances animals moved in 3 different treatments. As a first step I compared the central tendency with an ANOVA. I think that should be ok.

As a second step I would like to test if the frequency distributions of the distances moved are the same in all three treatments. For this I chose the “Two-Sample Kolmogorov-Smirnov Test”. The name indicates already that this is a test for only two samples. Since I have three treatments I did the test three times and adjusted the 0.05 alpha level with the Bonferroni method.

Now my questions:
Is there any possibility to calculate a new p-value for every comparison? (instead of adjusting the customarily used 0.05 level?)
Apparently there is a discussion ongoing if there are better methods than the Bonferroni, does anybody have a short (easily understandable) summary of this discussion and the conclusion until now?

Or maybe I should in general use another test instead of the “Two-Sample Kolmogorov-Smirnov Test” to compare the distributions that might help to avoid a post multiple comparison method? If yes, which?

Thanks so much in advance for your help!



TS Contributor
With only three treatment groups, it's overly conservative to adjust the alpha levels with a Bonferroni method. You could do 3 KS tests without adjusting alpha, in my opinion.

It's also been argued that the nonparametric version of the 1-way ANOVA, called the Kruskal-Wallis test (which many call a "test of whether or not the medians of >2 groups are the same") is actually a more general test, in that it is comparing distributions rather than medians.

If this test turns out to be significant, then you could follow up with Mann-Whitney U-tests (nonparametric analogues of the independent-samples t-test) - the same argument applies - with the U test, some would argue that it is a general distributional comparison and not necessarily a test between two medians.

With only 3 treatment levels, you shouldn't worry too much about adjusting the alpha level. If the ANOVA F-statistic or any other "omnibus" test is significant, then go ahead and do the three post-hoc comparisons. With only 3 treatment groups, there is little risk in an increasing Type I error rate.