Permutation test with skewed data

hlsmith

Not a robit
#1
Below are my data - small samples (n=22; n=14). The gaps in the histograms are just due to small sample sizes. In the parent population, data are normal looking with a very long stretched out right-tail. (EDIT: Of note, distributions are left-side bounded by zero)

What are the pros and cons to using a permutation test looking at mean differences? Should I do this with medians instead? Also, what are alternatives beyond Wilcoxon-style tests?

1525790985353.png

Here is the output from the permutation test with 500,000 iterations (doesn't seem too bad), vertical reference line represents the observed mean difference. Any suggestions on written interpretation of results that could be used by lay-person? Thx. EDIT: "Probability of a difference in means between groups at least as extreme as the observed difference given the null hypothesis is true."

TS Permutation SGPlot3.png
 
Last edited:

hlsmith

Not a robit
#2
Here is the median version (a little gappy given the limited number of differences available):

TS Permutation SGPlot3.png

For additional reference: given a null hypothesis you would see the mean difference or larger 1% of the time and the median difference or larger 14% of the time.

In my mind I am favoring the later approach. @Dason - do you have a possible Bayesian approach suggestion that I could use given the data distributions. I was examining the Kruscke's BEST (Bayesian Estimate Superior to T-test), which does alright with outliers - though I imagine it is not ideal for this type of skewness.
 

hlsmith

Not a robit
#3
Also, I would be open to Poisson models. In general I am looking for good fit/appropriateness along with interpretable outcomes that can be disseminated to none statistically savvy individuals.
 
Last edited:

hlsmith

Not a robit
#4
Well I just explored a likely better approach. I will need to check assumptions, but below is my quantile regression output. Showing comparable estimates between groups for outcome variable (duration), with durations getting larger in the upper quants and separation occurring in the last group. The influence of the small sample size can be seen in the graph.

TS Permutation SGPlot3.png
 
Last edited:
#5
Wilcoxon-Mann-Whitney (WMW) are sensitive to "spread" and I believe that it is also sensitive to skewness. A few weeks ago I thought of suggesting to replace the "standard" WMW with a permutation test. Then I learned that the WMW test is a special case of a permutations test. So I guess that the permutations test has the same disadvantages as the WMW test. As you have seen I refer to the papers by Fagerland Sandvik about WMW.

Byt what happens when you try a skewed distribution? Try lognormal, gamma and Weibull. (I think that they are called a generalized gamma family, but I can't find any source for that.)
 

hlsmith

Not a robit
#6
@GretaGarbo

Yes, given my small sample I did previously run an exact Wilcoxon rank sum test of median based on Monte Carlo. It gave significant results (even though that is not what I am fishing for ~ I want to define the data generating process and whether sample group comes into play), but it is not that interpretable like the permutation test. I end up with a probability estimate based on null hypothesis testing.

I am glad I found a use for the quantile regression. Which provides a representation of the data distributions. Though, I have concerns (e.g., false discovery and alpha corrections) when it runs multiple tests at prespecified quants. See my post at on cross-validated: https://stats.stackexchange.com/questions/345150/quantile-regression-and-false-discovery.

Yes, the gamma distribution also came to my mind along with perhaps some type of generalized estimating equation. But I am still not versed enough to knowingly/accurately go that direction. Do I need to specify moment equivalents for the assumed underlying distribution, etc.?
 
#7
Well, for the lognormal it is of course just to take logs and look at the distribution with a histogram and - if it looks somewhat normal, I often do a qqplot too - then do a t-test on the logged values.

The gamma can be estimated with a generalized linear model. Just specify family=Gamma.
Example:

Code:
#n=22; n=14

y <- c(rgamma(22, shape = 1, rate = 1/3), rgamma(14, shape = 1, rate = 1/6))
group <- c(rep(1,22), rep(2,14))
group
hist(y[1:22])
hist(y[23:36])
boxplot(y ~ group )
mean(y[1:22])
mean(y[23:36])

summary(glm(y ~ as.factor(group), family = Gamma(link = "log") ))
 

hlsmith

Not a robit
#8
My good friend @GretaGarbo - you are specifying values for the distribution. That is where I am not yet savvy. Seems similar to say when you do Bayesian Poisson regression, which is on my list of things for the next year (i.e., Bayesian analyses), but not yet inside my head. I slightly recall that the pre-specifications are the rate or prevalence, etc. Can you provide more details please.

Here is a toy set we can use. I will be offline for awhile, but will check back later when I am available. Of note these are integers representing rounded time to days.

G1 = {2, 6, 11, 14, 3, 4, 9, 1, 4, 5, 2, 7, 7, 2};

G2 = {4, 7, 1, 3, 3, 3, 0, 1, 4, 2, 5, 5, 2, 4, 0, 4, 3, 1, 5, 2, 3, 2};
 
Last edited:
#11
Follow-up, jamming toy data into @GretaGarbo code:

Code:
NB <- read.table(header = TRUE, text =
               "days group
                2 a
                6 a
                11 a
                14 a
                3 a
                4 a
                9 a
                1 a
                4 a
                5 a
                2 a
                7 a
                7 a
                2 a
                4 b
                7 b
                1 b
                3 b
                3 b
                3 b
                0.5 b
                1 b
                4 b
                2 b
                5 b
                5 b
                2 b
                4 b
                0.5 b
                4 b
                3 b
                1 b
                5 b
                2 b
                3 b
                2 b")
hist(NB$days[1:14])
hist(NB$days[15:36])
boxplot(NB$days ~ NB$group )
mean(NB$days[1:14])
mean(NB$days[15:36])

summary(glm(NB$days ~ as.factor(NB$group),  family = Gamma(link = "log")))
P.S., Got following warning and turned "0" into "0.5" just to get code to run in this example.
"non-positive values not allowed for the 'gamma' family"
 
#12
Since the data are continuous data originally, that have been rounded, you could think of the zero value as a censored value.

If the original data was between 0 and 0.5, but then it has been rounded to 0, you can think of it as censored below 0.5.

If you have two values y1 and y2, you can write the likelihood as: L(mu) = f(y1)*f(y2).

But if you also have a censored value y0 below 0.5 you can write the likelihood as: L(mu) = f(y1)*f(y2)*F(0.5)

where F() is the distribution function F(0.5) = P(Y0 <= 0.5) and f() is the density. Then you can maximize the likelihood by trying different values for mu.

Actually all rounded values are interval censored.

There is an R package called NADA that can estimate censored values.
 
#13
Wilcoxon-Mann-Whitney (WMW) are sensitive to "spread" and I believe that it is also sensitive to skewness. A few weeks ago I thought of suggesting to replace the "standard" WMW with a permutation test. Then I learned that the WMW test is a special case of a permutations test. So I guess that the permutations test has the same disadvantages as the WMW test. As you have seen I refer to the papers by Fagerland Sandvik about WMW.

Byt what happens when you try a skewed distribution? Try lognormal, gamma and Weibull. (I think that they are called a generalized gamma family, but I can't find any source for that.)
Hi Greta :)

Per my understanding, you can use the Mann-Whitney U test for any distribution, include a skew one, the only question is what will the test check?

1. When the two groups don't have a similar shape, the test will compare value from the entire distribution say the test will compare a random value from one group to a random value from the other group.
2. If the two groups have a similar distribution curve, the test will also compare the medians of the two groups.
3. When both groups are symmetrical, the medians and the means will be similar.

I assume option 1 is as good as 2 or 3 since generally when we compare groups we really mean to the broader definition of comparing the entire groups (option 1) but it is much easier to simplify and restrict it to only one measurement, usually the average.
 
#14
Per my understanding, you can use the Mann-Whitney U test for any distribution, include a skew one, the only question is what will the test check?
I am sorry but I don't understand exactly what Obh mean. My point was that when two data sets have different "spread" (but the same mean) then the WMW does not have the promised quality of an error rate of 0.05. The error rate will be larger. And I believe that most people interpret a significant WMW as a difference in location.
 
#15
Hi Greta,

Sorry that you didn't understand my response. I always like to read your valued answers and appreciate your knowledge.
I agree with your current response, If both distributions have the same shape (even skewed shape) you can interrupt the test as a difference in a location which is also the difference in the medians.

But even if the shape is not similar you can show a significant result that the rank of one group is higher than the rank of the second group.
And I believe that this finding is important as well despite the fact that you can't claim for a significant difference in the medians.

For example, saying one group has a significantly bigger rank may say that if you take medicine1 the possibility of better recovery is higher than if you take medicine2, and this means a lot. won't you take medicine1? ;)