I wasn't sure on that either. But I do think the ratio is the better measure. Pretty much anytime you're dealing with scale you want to deal with ratios.
I'm sorry, but I really don't know what you're saying...
855 of the 10000 resampled variance differences are either greater than the actual variance difference or less than the actual variance difference*-1.
153 of the 10000 resampled variance ratios are either greater than the actual variance ratio or less than the actual variance ratio^-1.
In what sense does this indicate "agreement" between the variance difference and the variance ratio?
I wasn't sure on that either. But I do think the ratio is the better measure. Pretty much anytime you're dealing with scale you want to deal with ratios.
Any specific thoughts on why it is the better measure here? The answers given by the two clearly differ somewhat. But it's not at all clear which answer I should "trust" more.
I mean, let's say we were interested in assessing through bootstrapping the relative means (a more typical example) rather than the relative variances. Rather than computing the mean difference at each resample, we could in principle compute the mean ratio and get a distribution of mean ratios centered at 1 just like we did for the variance ratio above. How would this be an obviously better or worse approach than resampling the mean difference? Isn't one of the points of bootstrapping that you don't have to make assumptions about the distribution of the statistic you are interested in?
Were you constructing the bootstrap confidence intervals using the percentile-based method? If so, I'm initially surprised at your results. It seems that the subtraction of one number from another is monotonically related to the ratio (1/2>1/4 as (1-2)>(1-4). Therefore, it seems that, if the same bootstraps are used via the same seed, that the p-values would be the same because the ranks should be the same. One possibility is a rounding error being more common in one scenario. I may be way off here however.
Thanks everyone for the discussion so far. I'll will read more carefully what has come before and post again soon.
Yes, the p-values and confidence intervals are based on quantiles of the actual resampled statistics. I was surprised too.
the confidence interval for the difference includes 0, not rejct. The conf incterval for the ratio includes unity, not reject. Second, the calculation of the p-value is wrong.
Masteras, the confidence intervals are around the null hypothesis. So of course they contain 0 and 1. In fact, you may notice that they are centered at 0 and 1. We reject if the actual point estimate is within the confidence interval around the null. For the variance difference, it is within that interval. For the variance ratio, it is not.
The p-values are correct. They are two-tailed p-values, so they are calculated as the proportion of bootstrap-resampled statistics that are either greater than the actual point estimate OR less than the point estimate IF the order of the groups had been reversed. Do you think these should be calculated some other way?
quantile(results$t[,2],probs=c(.025,.5,.975))
2.5% 50% 97.5%
0.6216802 0.9999918 1.6083679 , I see unity is icluded in the ratio.
"We reject if the actual point estimate is within the confidence interval around the null." reject?
The p-value for the bootstrap test is (#Tboot>Tobs+1)(B+1), where B is the number of bootrstrap samples.
Looking at the two histograms, the ratio histogram is truncated at zero and so the way that numbers are handled in terms of rounding would be more likely to make a difference. It would be interesting to run this again but make the ratio double precision and use a smaller number of replicates, so that they could be examined in a spreadsheet for any "weirdness". Also, taking the log of the ratio may accomplish the same thing. As it stands though, the ratio and differences should have the same ranks, so I wonder if we are viewing an artifact ?!?
I have R on my other machine, I'll try what I suggested later this evening.
Seems I was off my rocker on the last few posts. The ranks will vary between differences and ranks as shown by a simple example (6-4=4-2 while 6/4 < 4/2). For now, I've gone with differences. I'll change if it gets "sent back". Thanks again. Will post anything I learn that is informative to this topic.
Tweet |