mann whitney test or Kolmogorov smirnov test?

[Marga]

I'm working with non parametric methods to compare two samples and detect diferences between them. I used the mann-whitney test and obtained, for all the variables, p values less than 0,05 but when i used the KS test i found, for two of the four variables, p values with more than 0.05.
I would like to know which test is better? Or there some problem that i'm not able to capture?
Thank for any help?:tup:
Marge.

[JohnM]

These two tests serve entirely different purposes, so I'm not sure what you're trying to do. The Mann-Whitney is a nonparametric version of the t-test, checking for a significant difference in group medians. The K-S test is a goodness-of-fit test (i.e., is the sample distribution a close approximation to the normal distribution).

[Marga]

These two tests serve entirely different purposes, so I'm not sure what you're trying to do. The Mann-Whitney is a nonparametric version of the t-test, checking for a significant difference in group medians. The K-S test is a goodness-of-fit test (i.e., is the sample distribution a close approximation to the normal distribution).
Hi!
The KS test i refer in the initial question is the ks test for two independent samples. I wish to know if there exists differences between two samples. In SPSS, for two independent samples, i have choosen the mann-whitney test (location) and the KS test (shape) but for the first test all the variables have pvalues less than 0.05 and for the second test just two variables presents pvalues less than 0.05.

[JohnM]

OK - it simply means that for all the variables, the two groups differ on location, but only 2 of them differ on shape.

[Marga]

THANKS!
It´s really good to know that there's somebody to help!
So i think i can use both tests and no one is preferable to another, or not?

[JohnM]

You can use both tests, but keep in mind that they test for different things, one is for location, the other is for shape.

[elnaz]

Hello
sorry , your answer is not true, please attend that k-s test is for testing normality but mann whitney is for comparing two independent population.

[JohnM]

Yes, it is true. K-S can be used to compare the shapes of two distributions - here's a link that proves it.

http://www.physics.csbsju.edu/stats/KS-test.html

[elnaz]

hello
your speeches is true if distribution of population exactly be known ,
but you know there is many density function that we dont know them , we know e.g : normal ,gamma,poisson , ...
and k-s test do test for famouse distribution , but in campare mean problems in nonparametric we dont know about type of density ,that is free of distribution, then when we dont know about type of distribution , we should do nonparametric test not k-s test(we do k-s for limitary distribution )

[JohnM]

I'm sorry, but I have to disagree.

[elnaz]

no no
no problem

[statstats]

According to Siegel (1956), "the two-tailed test is sensitive to any kind of difference in the distributions from which the two samples were drawn" (p. 126).

The test compares the cumulative frequencies of the two samples. If there is a significant difference at any point, then the null hypothesis will be rejected.

The M-W test focuses on central tendency while the K-S is an omnibus test. Like JohnM wrote, the test will be appropriate depending on your research question(s).

Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.

[Marga]

Thanks for the reference!
Really what i was trying to demonstrate is that it exists differences between two small groups, regarding performance variables. One assumption of the Mann-whitney test is that the two distributions must be similar in shape. According instructions of the SPSS, the KS test for two independent samples can be useful to validate or not that assumption.
Am I right?
Thanks for your help,
Margue

[statstats]

In the British Medical Journal, Hart (2001) states that "the Mann-Whitney test is a test of both location and shape" (see: http://www.bmj.com/cgi/content/full/323/7309/391).

For complicating the matter, the K-S tests both difference in size, dispersion and in central tendency. Some call it an omnibus test, because it compares different aspects of the distributions of interest. "The K-S test [...] is sensitive to any kind of distributional difference" (Sheskin, 2004, p. 453). This means central tendency, variability, skewness and kurtosis.

So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test.

I wonder if an exploratory analysis of the shape, along with reporting skewness and kurtosis could be sufficient to verify the assumption of similarity between distributions.

[Marga]

Thanks! i use the box plots to explore tha shape as well as some descriptive statistics, as you recomend!!!:yup:

[LaMaskin]

"So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test."

Question: How can I increase the power of a two-sample K-S test? I need to minimise type II error....

[LaMaskin]

So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test.

How do I increase the power of a two-sample KS test?

[ugulino]

Hey People,

I've found this forum amazing. Can some of you help me?

Let me explain:

I have two independent samples. I wanna test if these 2 samples are significantly different. One sample was taken from grades given to a process by under-graduate students. The other sample was taken from grades given by post-graduate students to the same process.

I wanna know if under-graduate and post-graduate see the same process in different ways. The size of the samples is small (13 observations for the under-graduate students, and 14 observations for the post-graduate students).

Some descriptive statistics: same median, different means (under-graduate mean is 4.2, and post-graduate mean is 3.6).

With M-W, I've found p-value < 0.0001
With K-S, I've found p-value = 0.064 (6,4&#37;, it doesn't pass to the alpha of 5%)

What can I conclude from that? The samples have different location but same distribution? In other words, the first group (under-graduate) liked more the process than the second group, but they use to give grades in the same way (distribution) ?

It's kind a though analysis to me. Can someone help me?

PS.: I'm using XLStat

[ugulino]

According to Siegel & Castellan (2006, Non-parametrical Statistics to behavioral sciences), a two-sided Kolmogorov-Smirnov test is sensitive to any kind of difference in the distribution of two samples: differences on location (central tendency), dispersion, asymmetry, etc.

So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov and I can be confident to say that these 2 samples have the same distribution (p-value = 6.4&#37;, alpha-value = 5%). There are no differences in the way these two groups see the same process. Am I right?


Thanks.

[Mean Joe]

So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov

I think it should be said that you are not restricted to use only K-S.

I think that the K-S test often will not reject the null hypothesis when the sample sizes are small. The K-S test is a nice little test that measures the difference in the empirical cdf's (a pretty common sense thing to measure, if you want to see if two distributions are the same). And the critical value for the test is proportional to 1/sqrt(n)--my book says it is equal to 1.36/sqrt(n).

Regardless, a significant difference would be detected by the K-S test if the difference in the pdf's is greater than 0.25 (using n=14), eg. at least 4 more people in the undergrads giving a 1 than the graduates, or 4 more graduates giving a cumulative 1/2/3 than the undergraduates.

So keep that in mind, that the K-S rarely rejects the null hypothesis in small samples, and your p-value is .064, quite close to the magical threshold. And you have another test that is giving a significantly small p-value.

[ugulino]

I think it should be said that you are not restricted to use only K-S.
(...)
So keep that in mind, that the K-S rarely rejects the null hypothesis in small samples, and your p-value is .064, quite close to the magical threshold. And you have another test that is giving a significantly small p-value.

Well, in my research, reject the null hypothesis is a more conservative approach. I'm afraid to be biased to say that these two samples have similar distribution (not significantly different, accepting the null hypothesis), because if I can really say that they are similar, then it means to me that doesn't matter what of these two groups I'm analyzing, because they see the process in the very same way. So I can analyse the grades of these two groups as it was only one big group.

That's why I'm concerned to get the best decision. I don't want to get a biased result, but at the same time I don't want to say that these two results are significantly different without the sure that they really are.

What would you do in my place?

Thanks


Ugulino

[wipeout]

Maybe you should read some articles about criticism of hypothesis testing and statistical significance v/s practical significance.

As squareandrare said in another post: "any difference between the means will cause the null hypothesis to be rejected for a large enough sample size".

With a enough sample size you can ALWAYS prove anything, you can always find a "difference" between method A and Method B, the important question is:

Does this observed difference have practical importance?

Many statistician are using confidence interval insted of hypothesis testing.

[TheEcologist]

So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov and I can be confident to say that these 2 samples have the same distribution (p-value = 6.4&#37;, alpha-value = 5%). There are no differences in the way these two groups see the same process. Am I right?


Watch out for what is known as a 'logical fallacy'. Lack of evidence for the alternative hypothesis should never be viewed as evidence for the null. You can not state that a hypothesis is true only because it has not been proven false.

I dont know what you want to do with your results but if for publication or anything peer reviewed you might get a nasty review if your critic smells such a thing out. If its a thesis paper, also watch out.

So basically watch out for 'negative evidence' logic, it is generally frowned upon - people also call it an 'Argument from ignorance':
http://en.wikipedia.org/wiki/Argument_from_ignorance

here's another example of this:
http://faculty.vassar.edu/lowry/ch8pt3.html

What to do:

You cant say with confidence that these 2 samples come from the same distribution. You can only say that there is little evidence that they are different and usually people are then willing to accept that you assume that they are the same in further analysis. Just don't fall into this well known pitfall.

[TheEcologist]

Hey People,
What can I conclude from that? The samples have different location but same distribution? In other words, the first group (under-graduate) liked more the process than the second group, but they use to give grades in the same way (distribution) ?


Here's a good way to see if your conclusions are plausible, subtract the median of group A from all observations in group A and then do the same for group B (using group B median for group B obs!). - In effect you've now removed the different locations, making the central tendency for both datasets zero - If your conclusions are correct, you've now removed the sole source of difference between the groups.. so now recalculate your KS and post your results.

Basically, if the p-value increases your conclusions are plausible - I suspect it will.