# Permutation test with skewed data

#### hlsmith

##### Not a robit
Below are my data - small samples (n=22; n=14). The gaps in the histograms are just due to small sample sizes. In the parent population, data are normal looking with a very long stretched out right-tail. (EDIT: Of note, distributions are left-side bounded by zero)

What are the pros and cons to using a permutation test looking at mean differences? Should I do this with medians instead? Also, what are alternatives beyond Wilcoxon-style tests?

Here is the output from the permutation test with 500,000 iterations (doesn't seem too bad), vertical reference line represents the observed mean difference. Any suggestions on written interpretation of results that could be used by lay-person? Thx. EDIT: "Probability of a difference in means between groups at least as extreme as the observed difference given the null hypothesis is true."

Last edited:

#### hlsmith

##### Not a robit
Here is the median version (a little gappy given the limited number of differences available):

For additional reference: given a null hypothesis you would see the mean difference or larger 1% of the time and the median difference or larger 14% of the time.

In my mind I am favoring the later approach. @Dason - do you have a possible Bayesian approach suggestion that I could use given the data distributions. I was examining the Kruscke's BEST (Bayesian Estimate Superior to T-test), which does alright with outliers - though I imagine it is not ideal for this type of skewness.

#### hlsmith

##### Not a robit
Also, I would be open to Poisson models. In general I am looking for good fit/appropriateness along with interpretable outcomes that can be disseminated to none statistically savvy individuals.

Last edited:

#### hlsmith

##### Not a robit
Well I just explored a likely better approach. I will need to check assumptions, but below is my quantile regression output. Showing comparable estimates between groups for outcome variable (duration), with durations getting larger in the upper quants and separation occurring in the last group. The influence of the small sample size can be seen in the graph.

Last edited:

#### GretaGarbo

##### Human
Wilcoxon-Mann-Whitney (WMW) are sensitive to "spread" and I believe that it is also sensitive to skewness. A few weeks ago I thought of suggesting to replace the "standard" WMW with a permutation test. Then I learned that the WMW test is a special case of a permutations test. So I guess that the permutations test has the same disadvantages as the WMW test. As you have seen I refer to the papers by Fagerland Sandvik about WMW.

Byt what happens when you try a skewed distribution? Try lognormal, gamma and Weibull. (I think that they are called a generalized gamma family, but I can't find any source for that.)

#### hlsmith

##### Not a robit
@GretaGarbo

Yes, given my small sample I did previously run an exact Wilcoxon rank sum test of median based on Monte Carlo. It gave significant results (even though that is not what I am fishing for ~ I want to define the data generating process and whether sample group comes into play), but it is not that interpretable like the permutation test. I end up with a probability estimate based on null hypothesis testing.

I am glad I found a use for the quantile regression. Which provides a representation of the data distributions. Though, I have concerns (e.g., false discovery and alpha corrections) when it runs multiple tests at prespecified quants. See my post at on cross-validated: https://stats.stackexchange.com/questions/345150/quantile-regression-and-false-discovery.

Yes, the gamma distribution also came to my mind along with perhaps some type of generalized estimating equation. But I am still not versed enough to knowingly/accurately go that direction. Do I need to specify moment equivalents for the assumed underlying distribution, etc.?

#### GretaGarbo

##### Human
Well, for the lognormal it is of course just to take logs and look at the distribution with a histogram and - if it looks somewhat normal, I often do a qqplot too - then do a t-test on the logged values.

The gamma can be estimated with a generalized linear model. Just specify family=Gamma.
Example:

Code:
#n=22; n=14

y <- c(rgamma(22, shape = 1, rate = 1/3), rgamma(14, shape = 1, rate = 1/6))
group <- c(rep(1,22), rep(2,14))
group
hist(y[1:22])
hist(y[23:36])
boxplot(y ~ group )
mean(y[1:22])
mean(y[23:36])

summary(glm(y ~ as.factor(group), family = Gamma(link = "log") ))

#### hlsmith

##### Not a robit
My good friend @GretaGarbo - you are specifying values for the distribution. That is where I am not yet savvy. Seems similar to say when you do Bayesian Poisson regression, which is on my list of things for the next year (i.e., Bayesian analyses), but not yet inside my head. I slightly recall that the pre-specifications are the rate or prevalence, etc. Can you provide more details please.

Here is a toy set we can use. I will be offline for awhile, but will check back later when I am available. Of note these are integers representing rounded time to days.

G1 = {2, 6, 11, 14, 3, 4, 9, 1, 4, 5, 2, 7, 7, 2};

G2 = {4, 7, 1, 3, 3, 3, 0, 1, 4, 2, 5, 5, 2, 4, 0, 4, 3, 1, 5, 2, 3, 2};

Last edited:

#### Dason

She was specifying values to create sample data. The actual model did the parameter estimation.

#### hlsmith

##### Not a robit
Yeah, I gotcha - pretty apparent now. I skimmed the post on my way out the office earlier.

#### hlsmith

##### Not a robit
Follow-up, jamming toy data into @GretaGarbo code:

Code:
NB <- read.table(header = TRUE, text =
"days group
2 a
6 a
11 a
14 a
3 a
4 a
9 a
1 a
4 a
5 a
2 a
7 a
7 a
2 a
4 b
7 b
1 b
3 b
3 b
3 b
0.5 b
1 b
4 b
2 b
5 b
5 b
2 b
4 b
0.5 b
4 b
3 b
1 b
5 b
2 b
3 b
2 b")
hist(NB$days[1:14]) hist(NB$days[15:36])
boxplot(NB$days ~ NB$group )
mean(NB$days[1:14]) mean(NB$days[15:36])

summary(glm(NB$days ~ as.factor(NB$group),  family = Gamma(link = "log")))
P.S., Got following warning and turned "0" into "0.5" just to get code to run in this example.
"non-positive values not allowed for the 'gamma' family"

#### GretaGarbo

##### Human
Since the data are continuous data originally, that have been rounded, you could think of the zero value as a censored value.

If the original data was between 0 and 0.5, but then it has been rounded to 0, you can think of it as censored below 0.5.

If you have two values y1 and y2, you can write the likelihood as: L(mu) = f(y1)*f(y2).

But if you also have a censored value y0 below 0.5 you can write the likelihood as: L(mu) = f(y1)*f(y2)*F(0.5)

where F() is the distribution function F(0.5) = P(Y0 <= 0.5) and f() is the density. Then you can maximize the likelihood by trying different values for mu.

Actually all rounded values are interval censored.

There is an R package called NADA that can estimate censored values.

#### obh

##### Member
Wilcoxon-Mann-Whitney (WMW) are sensitive to "spread" and I believe that it is also sensitive to skewness. A few weeks ago I thought of suggesting to replace the "standard" WMW with a permutation test. Then I learned that the WMW test is a special case of a permutations test. So I guess that the permutations test has the same disadvantages as the WMW test. As you have seen I refer to the papers by Fagerland Sandvik about WMW.

Byt what happens when you try a skewed distribution? Try lognormal, gamma and Weibull. (I think that they are called a generalized gamma family, but I can't find any source for that.)
Hi Greta

Per my understanding, you can use the Mann-Whitney U test for any distribution, include a skew one, the only question is what will the test check?

1. When the two groups don't have a similar shape, the test will compare value from the entire distribution say the test will compare a random value from one group to a random value from the other group.
2. If the two groups have a similar distribution curve, the test will also compare the medians of the two groups.
3. When both groups are symmetrical, the medians and the means will be similar.

I assume option 1 is as good as 2 or 3 since generally when we compare groups we really mean to the broader definition of comparing the entire groups (option 1) but it is much easier to simplify and restrict it to only one measurement, usually the average.

#### GretaGarbo

##### Human
Per my understanding, you can use the Mann-Whitney U test for any distribution, include a skew one, the only question is what will the test check?
I am sorry but I don't understand exactly what Obh mean. My point was that when two data sets have different "spread" (but the same mean) then the WMW does not have the promised quality of an error rate of 0.05. The error rate will be larger. And I believe that most people interpret a significant WMW as a difference in location.

Hi Greta,