+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 16 to 24 of 24

Thread: mann whitney test or Kolmogorov smirnov test?

  1. #16
    Points: 3,162, Level: 34
    Level completed: 75%, Points required for next Level: 38

    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    K-S test power




    "So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test."

    Question: How can I increase the power of a two-sample K-S test? I need to minimise type II error....

  2. #17
    Points: 3,162, Level: 34
    Level completed: 75%, Points required for next Level: 38

    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    So, the K-S doesn't test only the difference in shape. Furthermore, a nonsignificant result could be be caused by a low power. This is a cause of concern withe the K-S test.

    How do I increase the power of a two-sample KS test?

  3. #18
    Points: 1,639, Level: 23
    Level completed: 39%, Points required for next Level: 61

    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Mann-Whitney or K-S, same question, different research

    Hey People,

    I've found this forum amazing. Can some of you help me?

    Let me explain:

    I have two independent samples. I wanna test if these 2 samples are significantly different. One sample was taken from grades given to a process by under-graduate students. The other sample was taken from grades given by post-graduate students to the same process.

    I wanna know if under-graduate and post-graduate see the same process in different ways. The size of the samples is small (13 observations for the under-graduate students, and 14 observations for the post-graduate students).

    Some descriptive statistics: same median, different means (under-graduate mean is 4.2, and post-graduate mean is 3.6).

    With M-W, I've found p-value < 0.0001
    With K-S, I've found p-value = 0.064 (6,4&#37;, it doesn't pass to the alpha of 5%)

    What can I conclude from that? The samples have different location but same distribution? In other words, the first group (under-graduate) liked more the process than the second group, but they use to give grades in the same way (distribution) ?

    It's kind a though analysis to me. Can someone help me?

    PS.: I'm using XLStat
    Last edited by ugulino; 05-06-2010 at 07:32 PM. Reason: To inform that I'm using XLStat

  4. #19
    Points: 1,639, Level: 23
    Level completed: 39%, Points required for next Level: 61

    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Considering to use Kolmogorov-Smirnov

    According to Siegel & Castellan (2006, Non-parametrical Statistics to behavioral sciences), a two-sided Kolmogorov-Smirnov test is sensitive to any kind of difference in the distribution of two samples: differences on location (central tendency), dispersion, asymmetry, etc.

    So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov and I can be confident to say that these 2 samples have the same distribution (p-value = 6.4&#37;, alpha-value = 5%). There are no differences in the way these two groups see the same process. Am I right?


    Thanks.
    Last edited by ugulino; 05-06-2010 at 07:32 PM.

  5. #20
    TS Contributor
    Points: 12,501, Level: 73
    Level completed: 13%, Points required for next Level: 349

    Posts
    951
    Thanks
    0
    Thanked 103 Times in 100 Posts
    Quote Originally Posted by ugulino View Post
    So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov
    I think it should be said that you are not restricted to use only K-S.

    I think that the K-S test often will not reject the null hypothesis when the sample sizes are small. The K-S test is a nice little test that measures the difference in the empirical cdf's (a pretty common sense thing to measure, if you want to see if two distributions are the same). And the critical value for the test is proportional to 1/sqrt(n)--my book says it is equal to 1.36/sqrt(n).

    Regardless, a significant difference would be detected by the K-S test if the difference in the pdf's is greater than 0.25 (using n=14), eg. at least 4 more people in the undergrads giving a 1 than the graduates, or 4 more graduates giving a cumulative 1/2/3 than the undergraduates.

    So keep that in mind, that the K-S rarely rejects the null hypothesis in small samples, and your p-value is .064, quite close to the magical threshold. And you have another test that is giving a significantly small p-value.
    Last edited by Mean Joe; 05-07-2010 at 11:46 AM.

  6. #21
    Points: 1,639, Level: 23
    Level completed: 39%, Points required for next Level: 61

    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Mean Joe View Post
    I think it should be said that you are not restricted to use only K-S.
    (...)
    So keep that in mind, that the K-S rarely rejects the null hypothesis in small samples, and your p-value is .064, quite close to the magical threshold. And you have another test that is giving a significantly small p-value.
    Well, in my research, reject the null hypothesis is a more conservative approach. I'm afraid to be biased to say that these two samples have similar distribution (not significantly different, accepting the null hypothesis), because if I can really say that they are similar, then it means to me that doesn't matter what of these two groups I'm analyzing, because they see the process in the very same way. So I can analyse the grades of these two groups as it was only one big group.

    That's why I'm concerned to get the best decision. I don't want to get a biased result, but at the same time I don't want to say that these two results are significantly different without the sure that they really are.

    What would you do in my place?

    Thanks


    Ugulino

  7. #22
    Points: 7,346, Level: 56
    Level completed: 98%, Points required for next Level: 4

    Location
    Santiago, Chile
    Posts
    23
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Maybe you should read some articles about criticism of hypothesis testing and statistical significance v/s practical significance.

    As squareandrare said in another post: "any difference between the means will cause the null hypothesis to be rejected for a large enough sample size".

    With a enough sample size you can ALWAYS prove anything, you can always find a "difference" between method A and Method B, the important question is:

    Does this observed difference have practical importance?

    Many statistician are using confidence interval insted of hypothesis testing.

  8. #23
    R purist
    Points: 35,103, Level: 100
    Level completed: 0%, Points required for next Level: 0
    TheEcologist's Avatar
    Location
    United States
    Posts
    1,921
    Thanks
    303
    Thanked 607 Times in 341 Posts
    Quote Originally Posted by ugulino View Post
    So, based on this reference, it seems to me that I can use only Kolmogorov-Smirnov and I can be confident to say that these 2 samples have the same distribution (p-value = 6.4&#37;, alpha-value = 5%). There are no differences in the way these two groups see the same process. Am I right?
    Watch out for what is known as a 'logical fallacy'. Lack of evidence for the alternative hypothesis should never be viewed as evidence for the null. You can not state that a hypothesis is true only because it has not been proven false.

    I dont know what you want to do with your results but if for publication or anything peer reviewed you might get a nasty review if your critic smells such a thing out. If its a thesis paper, also watch out.

    So basically watch out for 'negative evidence' logic, it is generally frowned upon - people also call it an 'Argument from ignorance':
    http://en.wikipedia.org/wiki/Argument_from_ignorance

    here's another example of this:
    http://faculty.vassar.edu/lowry/ch8pt3.html

    What to do:

    You cant say with confidence that these 2 samples come from the same distribution. You can only say that there is little evidence that they are different and usually people are then willing to accept that you assume that they are the same in further analysis. Just don't fall into this well known pitfall.
    The true ideals of great philosophies always seem to get lost somewhere along the road..

  9. #24
    R purist
    Points: 35,103, Level: 100
    Level completed: 0%, Points required for next Level: 0
    TheEcologist's Avatar
    Location
    United States
    Posts
    1,921
    Thanks
    303
    Thanked 607 Times in 341 Posts

    Quote Originally Posted by ugulino View Post
    Hey People,
    What can I conclude from that? The samples have different location but same distribution? In other words, the first group (under-graduate) liked more the process than the second group, but they use to give grades in the same way (distribution) ?
    Here's a good way to see if your conclusions are plausible, subtract the median of group A from all observations in group A and then do the same for group B (using group B median for group B obs!). - In effect you've now removed the different locations, making the central tendency for both datasets zero - If your conclusions are correct, you've now removed the sole source of difference between the groups.. so now recalculate your KS and post your results.

    Basically, if the p-value increases your conclusions are plausible - I suspect it will.
    The true ideals of great philosophies always seem to get lost somewhere along the road..

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2

           




Similar Threads

  1. Using R for the Kolmogorov-Smirnov test
    By freebird2008 in forum R
    Replies: 13
    Last Post: 07-08-2012, 09:15 AM
  2. Kolmogorov smirnov test for goodness of fit
    By aledanda in forum Statistics
    Replies: 3
    Last Post: 08-03-2009, 04:34 PM
  3. Replies: 2
    Last Post: 08-05-2008, 11:22 AM
  4. On Kolmogorov-Smirnov test
    By mp83 in forum Statistics
    Replies: 0
    Last Post: 01-04-2008, 02:54 PM
  5. Replies: 1
    Last Post: 12-03-2007, 04:41 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats