Feedback on bootstrapping effect size and t-values

I am trying to learn to bootstrap per conversation here the other day.

I know there are packages to bootstrap but wanted to process more manually-ish. I have a made up data set of pre and post test scores where one outlier really pulls the scores down. Here's my questions:

Am I bootstrapping correctly (for both the ES and the t-value)

Re: Feedback on bootstrapping effect size and t-values

As mentioned prior, there are many many bootstrapping approaches. More common ones us the bootstrapping to calculate the precision (the confidence interval around the ES, which was calculated using your empirical data). Other common approaches that deal with your ES, will have you subtract each bootstrap calculated ES from the empirical ES and find the mean of these 10000, and add it for an optimistic bias correction. Not an R person, but these described approaches differ from yours I believe. There is also a corrected version of the first approach I described, but I don't recall its name - goes by three initials if I remember correctly. All readily available in R.

Re: Feedback on bootstrapping effect size and t-values

It looks like you're sampling the pre and post scores independently of one another. Typically you would sample the pairs together (so think about the "people" as the thing you're getting replicate samples from).

I don't have emotions and sometimes that makes me very sad.

Re: Feedback on bootstrapping effect size and t-values

Oh I get it Dason. Sample row indices and pass that so they stay paired. More like:

Code:

n <- nrow(diff_dat)
reps <- 10000
out <- vector(mode = "list", length = reps)
for (i in 1:reps){
ind <- sample(1:n, n, TRUE)
mod <- with(diff_dat,
t.test(Post[ind], Pre[ind], paired=TRUE)
)
out[[i]] <- list(t = mod[["statistic"]][[1]], es = es(mod))
}
dat <- data.frame(do.call(rbind, out))
dat[] <- lapply(dat, unlist)
apply(dat, 2, mean)

"If you torture the data long enough it will eventually confess."
-Ronald Harry Coase -

Re: Feedback on bootstrapping effect size and t-values

I like that example about bias correction you gave hlsmith. I've typically thought about bootstrapping a three step process

1. Process the representative replicate (sample w/replacement, block sample, pairs, etc.)
2. Compute the statistic on the replicates
3. Use the distribution of replicated statistics for inference

In the bias example, you're using the deviation of the replicate statistic from the empirical statistic to generate an average bias, correct? To me that seems more like typical simulations. It makes sense, though. Now I'm going to try and see the world through bootstrapping lenses so I don't miss a chance to practice this!

You should definitely use jQuery. It's really great and does all things.

Re: Feedback on bootstrapping effect size and t-values

I'd like an explanation of what you're trying to do instead of teasing it out of your code

1. Handle the paired replication as Dason said
2. Your computation looks fine
3. Are you just taking the average effect size from the replicate effect sizes? This seems more like a simulation to me, whereas bootstrapping you would use the empirical ES (which the replicate one should be very close, but if you're doing a bias correction, hls' approach would apply here) and enclose it in the bootstrapped confidence interval. So is what you're after the CI or the ES measurement?

You should definitely use jQuery. It's really great and does all things.