Thread: mann whitney test or Kolmogorov smirnov test?

1. K-S test power

"So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test."

Question: How can I increase the power of a two-sample K-S test? I need to minimise type II error....

2. So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test.

How do I increase the power of a two-sample KS test?

3. Mann-Whitney or K-S, same question, different research

Hey People,

I've found this forum amazing. Can some of you help me?

Let me explain:

I have two independent samples. I wanna test if these 2 samples are significantly different. One sample was taken from grades given to a process by under-graduate students. The other sample was taken from grades given by post-graduate students to the same process.

I wanna know if under-graduate and post-graduate see the same process in different ways. The size of the samples is small (13 observations for the under-graduate students, and 14 observations for the post-graduate students).

Some descriptive statistics: same median, different means (under-graduate mean is 4.2, and post-graduate mean is 3.6).

With M-W, I've found p-value < 0.0001
With K-S, I've found p-value = 0.064 (6,4&#37;, it doesn't pass to the alpha of 5%)

What can I conclude from that? The samples have different location but same distribution? In other words, the first group (under-graduate) liked more the process than the second group, but they use to give grades in the same way (distribution) ?

It's kind a though analysis to me. Can someone help me?

PS.: I'm using XLStat

4. Considering to use Kolmogorov-Smirnov

According to Siegel & Castellan (2006, Non-parametrical Statistics to behavioral sciences), a two-sided Kolmogorov-Smirnov test is sensitive to any kind of difference in the distribution of two samples: differences on location (central tendency), dispersion, asymmetry, etc.

So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov and I can be confident to say that these 2 samples have the same distribution (p-value = 6.4&#37;, alpha-value = 5%). There are no differences in the way these two groups see the same process. Am I right?

Thanks.

5. Originally Posted by ugulino
So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov
I think it should be said that you are not restricted to use only K-S.

I think that the K-S test often will not reject the null hypothesis when the sample sizes are small. The K-S test is a nice little test that measures the difference in the empirical cdf's (a pretty common sense thing to measure, if you want to see if two distributions are the same). And the critical value for the test is proportional to 1/sqrt(n)--my book says it is equal to 1.36/sqrt(n).

Regardless, a significant difference would be detected by the K-S test if the difference in the pdf's is greater than 0.25 (using n=14), eg. at least 4 more people in the undergrads giving a 1 than the graduates, or 4 more graduates giving a cumulative 1/2/3 than the undergraduates.

So keep that in mind, that the K-S rarely rejects the null hypothesis in small samples, and your p-value is .064, quite close to the magical threshold. And you have another test that is giving a significantly small p-value.

6. Originally Posted by Mean Joe
I think it should be said that you are not restricted to use only K-S.
(...)
So keep that in mind, that the K-S rarely rejects the null hypothesis in small samples, and your p-value is .064, quite close to the magical threshold. And you have another test that is giving a significantly small p-value.
Well, in my research, reject the null hypothesis is a more conservative approach. I'm afraid to be biased to say that these two samples have similar distribution (not significantly different, accepting the null hypothesis), because if I can really say that they are similar, then it means to me that doesn't matter what of these two groups I'm analyzing, because they see the process in the very same way. So I can analyse the grades of these two groups as it was only one big group.

That's why I'm concerned to get the best decision. I don't want to get a biased result, but at the same time I don't want to say that these two results are significantly different without the sure that they really are.

What would you do in my place?

Thanks

Ugulino

7. Maybe you should read some articles about criticism of hypothesis testing and statistical significance v/s practical significance.

As squareandrare said in another post: "any difference between the means will cause the null hypothesis to be rejected for a large enough sample size".

With a enough sample size you can ALWAYS prove anything, you can always find a "difference" between method A and Method B, the important question is:

Does this observed difference have practical importance?

Many statistician are using confidence interval insted of hypothesis testing.

8. Originally Posted by ugulino
So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov and I can be confident to say that these 2 samples have the same distribution (p-value = 6.4&#37;, alpha-value = 5%). There are no differences in the way these two groups see the same process. Am I right?
Watch out for what is known as a 'logical fallacy'. Lack of evidence for the alternative hypothesis should never be viewed as evidence for the null. You can not state that a hypothesis is true only because it has not been proven false.

I dont know what you want to do with your results but if for publication or anything peer reviewed you might get a nasty review if your critic smells such a thing out. If its a thesis paper, also watch out.

So basically watch out for 'negative evidence' logic, it is generally frowned upon - people also call it an 'Argument from ignorance':
http://en.wikipedia.org/wiki/Argument_from_ignorance

here's another example of this:
http://faculty.vassar.edu/lowry/ch8pt3.html

What to do:

You cant say with confidence that these 2 samples come from the same distribution. You can only say that there is little evidence that they are different and usually people are then willing to accept that you assume that they are the same in further analysis. Just don't fall into this well known pitfall.

9. Originally Posted by ugulino
Hey People,
What can I conclude from that? The samples have different location but same distribution? In other words, the first group (under-graduate) liked more the process than the second group, but they use to give grades in the same way (distribution) ?
Here's a good way to see if your conclusions are plausible, subtract the median of group A from all observations in group A and then do the same for group B (using group B median for group B obs!). - In effect you've now removed the different locations, making the central tendency for both datasets zero - If your conclusions are correct, you've now removed the sole source of difference between the groups.. so now recalculate your KS and post your results.

Basically, if the p-value increases your conclusions are plausible - I suspect it will.