# Thread: Feedback on bootstrapping effect size and t-values

1. ## Feedback on bootstrapping effect size and t-values

I am trying to learn to bootstrap per conversation here the other day.

I know there are packages to bootstrap but wanted to process more manually-ish. I have a made up data set of pre and post test scores where one outlier really pulls the scores down. Here's my questions:
1. Am I bootstrapping correctly (for both the ES and the t-value)
2. Do I have enough replications (I've read 10,000 is plenty)

I am calculating the ES for dependent samples using (from Andy Fields R book p. 393):

Code:
   Pre Post
1   71 81.0
2   76 86.0
3   59 63.0
4   71 71.0
5   77 79.0
6   80 79.0
7   68 70.0
8   92 91.0
9   81 83.0
10  63 65.0
11  88 90.0
12  70 37.5

Code:
diff_dat <- data.frame(
Pre = c(71, 76, 59, 71, 77, 80, 68, 92, 81, 63, 88, 70),
Post = c(81, 86, 63, 71, 79, 79, 70, 91, 83, 65, 90, 37.5)
)

es <- function(mod, digs = 3){ # p. 393 Andy Fields
t <- mod[["statistic"]][[1]]
df <- mod[["parameter"]][[1]]
r <- sqrt((t^2)/(t^2+df))
round(r, digs)
}

(mod <- with(diff_dat, t.test(Post, Pre, paired=TRUE)))

##         Paired t-test
##
## data:  Post and Pre
## t = -0.0133, df = 11, p-value = 0.9896
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -6.933515  6.850182
## sample estimates:
## mean of the differences
##             -0.04166667

es(mod)

## [1] 0.004

n <- nrow(diff_dat)

reps <- 10000
out <- vector(mode = "list", length = reps)

for (i in 1:reps){

mod <- with(diff_dat,
t.test(sample(Post, n, TRUE), sample(Pre, n, TRUE), paired=TRUE)
)
out[[i]] <- list(t = mod[["statistic"]][[1]], es = es(mod))
}

dat <- data.frame(do.call(rbind, out))
dat[] <- lapply(dat, unlist)
apply(dat, 2, mean)

##         t        es
## 0.1273034 0.2391871 

2. ## Re: Feedback on bootstrapping effect size and t-values

As mentioned prior, there are many many bootstrapping approaches. More common ones us the bootstrapping to calculate the precision (the confidence interval around the ES, which was calculated using your empirical data). Other common approaches that deal with your ES, will have you subtract each bootstrap calculated ES from the empirical ES and find the mean of these 10000, and add it for an optimistic bias correction. Not an R person, but these described approaches differ from yours I believe. There is also a corrected version of the first approach I described, but I don't recall its name - goes by three initials if I remember correctly. All readily available in R.

3. ## The Following 2 Users Say Thank You to hlsmith For This Useful Post:

bryangoodrich (04-28-2015), trinker (04-28-2015)

4. ## Re: Feedback on bootstrapping effect size and t-values

I liked that you used ES, I am slowly trying to incorporate more ES along with p-values.

5. ## Re: Feedback on bootstrapping effect size and t-values

It looks like you're sampling the pre and post scores independently of one another. Typically you would sample the pairs together (so think about the "people" as the thing you're getting replicate samples from).

6. ## The Following 2 Users Say Thank You to Dason For This Useful Post:

bryangoodrich (04-28-2015), trinker (04-28-2015)

7. ## Re: Feedback on bootstrapping effect size and t-values

True dat, Dason!

8. ## Re: Feedback on bootstrapping effect size and t-values

Dason how could I go about sampling them together?

9. ## Re: Feedback on bootstrapping effect size and t-values

Also would this give me the confidence intervals for the estimates?

Code:
quantile(dat[["t"]])[c(2, 4)]
quantile(dat[["es"]])[c(2, 4)]

10. ## Re: Feedback on bootstrapping effect size and t-values

Oh I get it Dason. Sample row indices and pass that so they stay paired. More like:

Code:
n <- nrow(diff_dat)

reps <- 10000
out <- vector(mode = "list", length = reps)

for (i in 1:reps){

ind <- sample(1:n, n, TRUE)
mod <- with(diff_dat,
t.test(Post[ind], Pre[ind], paired=TRUE)
)

out[[i]] <- list(t = mod[["statistic"]][[1]], es = es(mod))
}

dat <- data.frame(do.call(rbind, out))
dat[] <- lapply(dat, unlist)
apply(dat, 2, mean)

11. ## Re: Feedback on bootstrapping effect size and t-values

Can someone answer the confint question now?

12. ## Re: Feedback on bootstrapping effect size and t-values

I think this is the correct way from what I read:

Code:
quantile(dat[["t"]], c(.025,.975))
quantile(dat[["es"]], c(.025,.975))

13. ## Re: Feedback on bootstrapping effect size and t-values

Looks right for percentile intervals.

14. ## The Following User Says Thank You to hlsmith For This Useful Post:

trinker (04-28-2015)

15. ## Re: Feedback on bootstrapping effect size and t-values

Originally Posted by trinker
I think this is the correct way from what I read:

Code:
quantile(dat[["t"]], c(.025,.975))
quantile(dat[["es"]], c(.025,.975))
this is what i've seen as well (for a 2-sided hypothesis with alpha = .05)

16. ## The Following User Says Thank You to spunky For This Useful Post:

trinker (04-28-2015)

17. ## Re: Feedback on bootstrapping effect size and t-values

I like that example about bias correction you gave hlsmith. I've typically thought about bootstrapping a three step process

1. Process the representative replicate (sample w/replacement, block sample, pairs, etc.)
2. Compute the statistic on the replicates
3. Use the distribution of replicated statistics for inference

In the bias example, you're using the deviation of the replicate statistic from the empirical statistic to generate an average bias, correct? To me that seems more like typical simulations. It makes sense, though. Now I'm going to try and see the world through bootstrapping lenses so I don't miss a chance to practice this!

18. ## The Following User Says Thank You to bryangoodrich For This Useful Post:

trinker (04-28-2015)

19. ## Re: Feedback on bootstrapping effect size and t-values

@BG Yes I believe so.

Folks is it OK to bootstrap the effect size in this way as well?

20. ## Re: Feedback on bootstrapping effect size and t-values

I'd like an explanation of what you're trying to do instead of teasing it out of your code

1. Handle the paired replication as Dason said
3. Are you just taking the average effect size from the replicate effect sizes? This seems more like a simulation to me, whereas bootstrapping you would use the empirical ES (which the replicate one should be very close, but if you're doing a bias correction, hls' approach would apply here) and enclose it in the bootstrapped confidence interval. So is what you're after the CI or the ES measurement?

Page 1 of 3 1 2 3 Last

 Tweet

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts