Finding the p-value for two samples which are not normally distributed.

#1
Hello,

I am doing medical research and I have gathered data for two groups of patients with different characteristics. The data for each of the groups is not normally distributed. Let's say the null hypothesis is that there are no differences between the two groups, how can I come up with the p-value when the data is not normally distributed?

Thank you!
 

Englund

TS Contributor
#2
You could try to take the natural logarithms of the data and see if gets normally distributed, and then conduct a parametric test (i.e an Aspin-Welch two sample t-test). Otherwise, you could make use of one of all the non-parametric tests out there. If it is appropriate to use a parametric test, you should do that since they often (always?) have higher power.

Edit: If your samples are sufficiently large, asymptotical theory states that you can use two sample t-tests since this test statistica is normally distributed then...
 

Jake

Cookie Scientist
#3
Like Englund said, the relevant sampling distribution might still be fine. But if you want to be safe you could use a nonparametric bootstrap or permutation test.
 
#4
As I remember it, when the data is normally distributed then the parametric t-test has a few percentage points higher power (or at least efficiency) than the Mann-Whitney-Wilcoxon (MWW) test. So then they are essentially the same.

But if the distribution is non-normal then for specific distributions MWW can have considerably higher power that the t-test. (To my surprise!)

But MWW is “sensitive to spread”.

So even if two distributions have the same median but have different “spread” e.g. different standard deviation, MWW can give many significances, far more than 5%.

I have heard it been said, although I have nothing written to point to, that the t-test can be better in such situations and “be more non-parametric than MWW”, so to speak.

I seldom use MWW.

I would probably, as Jake suggested above, use a permutation test.
(Or like Englund suggested, try to use a transformation to try to get it more normally distributed.)
(Or expand the potential distributions to the exponential family (where the gamma distribution is similar to the above suggested lognormal distribution) and use a generalized linear model.)
(Or expand it further to the gamlss system...)
 
#5
thanks for the replies so far. I would also like to ask, would a non-parametric test like the two-sample Kolmogorov-Smirnov test also do the job? But its disadvantage is that it has low power?
 
#6
KS test compares the shapes of the distributions. I don't think it would be the best option for you. Instead try a Mann-Whitney U test, if your patients in the two groups are not matched. Otherwise, if each patient in each group is matched with a patient in another group according to some characteristics such as age, gender, weight, etc., you should run a Wilcoxon's signed ranks test, instead. These two tests assess the null hypothesis "there are no differences between this specific characteristic of patients (e.g, their blood pressure) within the two groups".

The test power of these test might be slightly lower than the parametric alternatives, but not too much lower. Their advantage is on the other hand that you are certain your test assumptions are met.

But still you might have a chance to run a parametric t-test (if your sample is large enough for example).
 
#7
KS test compares the shapes of the distributions. I don't think it would be the best option for you. Instead try a Mann-Whitney U test, if your patients in the two groups are not matched. Otherwise, if each patient in each group is matched with a patient in another group according to some characteristics such as age, gender, weight, etc., you should run a Wilcoxon's signed ranks test, instead. These two tests assess the null hypothesis "there are no differences between this specific characteristic of patients (e.g, their blood pressure) within the two groups".

The test power of these test might be slightly lower than the parametric alternatives, but not too much lower. Their advantage is on the other hand that you are certain your test assumptions are met.

But still you might have a chance to run a parametric t-test (if your sample is large enough for example).
Thank you for your reply. Mann-Whitney U test tells us more information. But I think it has an underlying assumption that the variances of the two samples are more or less the same which is not true with the data I have. Can I still apply this test here then?
 
#8
I would like to ask, what would be considered large. I have very skewed data actually. One sample has 50 patients, the other sample has around 400 patients.
 

CB

Super Moderator
#9
Thank you for your reply. Mann-Whitney U test tells us more information. But I think it has an underlying assumption that the variances of the two samples are more or less the same which is not true with the data I have. Can I still apply this test here then?
It depends on how you want to interpret the Mann-Whitney test. Lots of people try to interpret it as testing a difference in medians, or even means. For these interpretations to be valid you have to assume that both the shape as well as the scale (e.g., variance) of the two distributions are identical. IMO this assumption is very optimistic in most cases.

On the other hand, the more general interpretation of the Mann-Whitney test is that it tests this null hypothesis:

H0: the probability that a randomly drawn case from population X has a larger value on the outcome variable than a randomly drawn case from population Y is equal to 0.5.

You do not need to assume that the shape and scale distribution of the outcome variable is identical in the two groups for this interpretation to be valid.

See:
Fagerland, M. W., & Sandvik, L. (2009). The Wilcoxon–Mann–Whitney test under scrutiny. Statistics in Medicine, 28(10), 1487–1497. doi:10.1002/sim.3561