Comparing distributions, Two-Sample KS-test and Bonferroni

#1
Hi there,

The following is my problem for which I’m grateful to have any advice/opinions:
I measured the distances animals moved in 3 different treatments. As a first step I compared the central tendency with an ANOVA. I think that should be ok.

As a second step I would like to test if the frequency distributions of the distances moved are the same in all three treatments. For this I chose the “Two-Sample Kolmogorov-Smirnov Test”. The name indicates already that this is a test for only two samples. Since I have three treatments I did the test three times and adjusted the 0.05 alpha level with the Bonferroni method.

Now my questions:
Is there any possibility to calculate a new p-value for every comparison? (instead of adjusting the customarily used 0.05 level?)
Apparently there is a discussion ongoing if there are better methods than the Bonferroni, does anybody have a short (easily understandable) summary of this discussion and the conclusion until now?

Or maybe I should in general use another test instead of the “Two-Sample Kolmogorov-Smirnov Test” to compare the distributions that might help to avoid a post multiple comparison method? If yes, which?

Thanks so much in advance for your help!

Hilary
 

JohnM

TS Contributor
#2
With only three treatment groups, it's overly conservative to adjust the alpha levels with a Bonferroni method. You could do 3 KS tests without adjusting alpha, in my opinion.

It's also been argued that the nonparametric version of the 1-way ANOVA, called the Kruskal-Wallis test (which many call a "test of whether or not the medians of >2 groups are the same") is actually a more general test, in that it is comparing distributions rather than medians.

If this test turns out to be significant, then you could follow up with Mann-Whitney U-tests (nonparametric analogues of the independent-samples t-test) - the same argument applies - with the U test, some would argue that it is a general distributional comparison and not necessarily a test between two medians.

With only 3 treatment levels, you shouldn't worry too much about adjusting the alpha level. If the ANOVA F-statistic or any other "omnibus" test is significant, then go ahead and do the three post-hoc comparisons. With only 3 treatment groups, there is little risk in an increasing Type I error rate.