+ Reply to Thread
Page 1 of 3 1 2 3 LastLast
Results 1 to 15 of 32

Thread: Feedback on bootstrapping effect size and t-values

  1. #1
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Feedback on bootstrapping effect size and t-values




    I am trying to learn to bootstrap per conversation here the other day.

    I know there are packages to bootstrap but wanted to process more manually-ish. I have a made up data set of pre and post test scores where one outlier really pulls the scores down. Here's my questions:
    1. Am I bootstrapping correctly (for both the ES and the t-value)
    2. Do I have enough replications (I've read 10,000 is plenty)
    3. Should I be bootstrapping the CI instead/in addition to?
    4. Any additional feedback

    I am calculating the ES for dependent samples using (from Andy Fields R book p. 393):

    \sqrt{\frac{t^2}{t^2+df}}

    Code: 
       Pre Post
    1   71 81.0
    2   76 86.0
    3   59 63.0
    4   71 71.0
    5   77 79.0
    6   80 79.0
    7   68 70.0
    8   92 91.0
    9   81 83.0
    10  63 65.0
    11  88 90.0
    12  70 37.5

    Code: 
    diff_dat <- data.frame(
         Pre = c(71, 76, 59, 71, 77, 80, 68, 92, 81, 63, 88, 70), 
        Post = c(81, 86, 63, 71, 79, 79, 70, 91, 83, 65, 90, 37.5)
    )
    
    es <- function(mod, digs = 3){ # p. 393 Andy Fields
        t <- mod[["statistic"]][[1]]
        df <- mod[["parameter"]][[1]]
        r <- sqrt((t^2)/(t^2+df))
        round(r, digs)
    }
    
    (mod <- with(diff_dat, t.test(Post, Pre, paired=TRUE)))
    
    ##         Paired t-test
    ## 
    ## data:  Post and Pre
    ## t = -0.0133, df = 11, p-value = 0.9896
    ## alternative hypothesis: true difference in means is not equal to 0
    ## 95 percent confidence interval:
    ##  -6.933515  6.850182
    ## sample estimates:
    ## mean of the differences 
    ##             -0.04166667 
    
    es(mod)
    
    ## [1] 0.004
    
    n <- nrow(diff_dat)
    
    reps <- 10000
    out <- vector(mode = "list", length = reps)
    
    for (i in 1:reps){
    	
        mod <- with(diff_dat, 
            t.test(sample(Post, n, TRUE), sample(Pre, n, TRUE), paired=TRUE)
        )
        out[[i]] <- list(t = mod[["statistic"]][[1]], es = es(mod))
    }
    
    dat <- data.frame(do.call(rbind, out))
    dat[] <- lapply(dat, unlist)
    apply(dat, 2, mean)
    
    ##         t        es 
    ## 0.1273034 0.2391871 
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  2. #2
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Feedback on bootstrapping effect size and t-values

    As mentioned prior, there are many many bootstrapping approaches. More common ones us the bootstrapping to calculate the precision (the confidence interval around the ES, which was calculated using your empirical data). Other common approaches that deal with your ES, will have you subtract each bootstrap calculated ES from the empirical ES and find the mean of these 10000, and add it for an optimistic bias correction. Not an R person, but these described approaches differ from yours I believe. There is also a corrected version of the first approach I described, but I don't recall its name - goes by three initials if I remember correctly. All readily available in R.
    Stop cowardice, ban guns!

  3. The Following 2 Users Say Thank You to hlsmith For This Useful Post:

    bryangoodrich (04-28-2015), trinker (04-28-2015)

  4. #3
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Feedback on bootstrapping effect size and t-values

    I liked that you used ES, I am slowly trying to incorporate more ES along with p-values.
    Stop cowardice, ban guns!

  5. #4
    Devorador de queso
    Points: 95,866, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,936
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Feedback on bootstrapping effect size and t-values

    It looks like you're sampling the pre and post scores independently of one another. Typically you would sample the pairs together (so think about the "people" as the thing you're getting replicate samples from).
    I don't have emotions and sometimes that makes me very sad.

  6. The Following 2 Users Say Thank You to Dason For This Useful Post:

    bryangoodrich (04-28-2015), trinker (04-28-2015)

  7. #5
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Feedback on bootstrapping effect size and t-values

    True dat, Dason!
    Stop cowardice, ban guns!

  8. #6
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: Feedback on bootstrapping effect size and t-values

    Dason how could I go about sampling them together?
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  9. #7
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: Feedback on bootstrapping effect size and t-values

    Also would this give me the confidence intervals for the estimates?

    Code: 
    quantile(dat[["t"]])[c(2, 4)]
    quantile(dat[["es"]])[c(2, 4)]
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  10. #8
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: Feedback on bootstrapping effect size and t-values

    Oh I get it Dason. Sample row indices and pass that so they stay paired. More like:

    Code: 
    n <- nrow(diff_dat)
    
    reps <- 10000
    out <- vector(mode = "list", length = reps)
    
    for (i in 1:reps){
    	
        ind <- sample(1:n, n, TRUE)
        mod <- with(diff_dat, 
            t.test(Post[ind], Pre[ind], paired=TRUE)
        )
        
        out[[i]] <- list(t = mod[["statistic"]][[1]], es = es(mod))
    }
    
    dat <- data.frame(do.call(rbind, out))
    dat[] <- lapply(dat, unlist)
    apply(dat, 2, mean)
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  11. #9
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: Feedback on bootstrapping effect size and t-values

    Can someone answer the confint question now?
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  12. #10
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: Feedback on bootstrapping effect size and t-values

    I think this is the correct way from what I read:

    Code: 
    quantile(dat[["t"]], c(.025,.975))
    quantile(dat[["es"]], c(.025,.975))
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  13. #11
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Feedback on bootstrapping effect size and t-values

    Looks right for percentile intervals.
    Stop cowardice, ban guns!

  14. The Following User Says Thank You to hlsmith For This Useful Post:

    trinker (04-28-2015)

  15. #12
    TS Contributor
    Points: 22,423, Level: 93
    Level completed: 8%, Points required for next Level: 927
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Feedback on bootstrapping effect size and t-values

    Quote Originally Posted by trinker View Post
    I think this is the correct way from what I read:

    Code: 
    quantile(dat[["t"]], c(.025,.975))
    quantile(dat[["es"]], c(.025,.975))
    this is what i've seen as well (for a 2-sided hypothesis with alpha = .05)
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  16. The Following User Says Thank You to spunky For This Useful Post:

    trinker (04-28-2015)

  17. #13
    Probably A Mammal
    Points: 31,087, Level: 100
    Level completed: 0%, Points required for next Level: 0
    bryangoodrich's Avatar
    Location
    Sacramento, California, United States
    Posts
    2,564
    Thanks
    398
    Thanked 618 Times in 551 Posts

    Re: Feedback on bootstrapping effect size and t-values

    I like that example about bias correction you gave hlsmith. I've typically thought about bootstrapping a three step process

    1. Process the representative replicate (sample w/replacement, block sample, pairs, etc.)
    2. Compute the statistic on the replicates
    3. Use the distribution of replicated statistics for inference

    In the bias example, you're using the deviation of the replicate statistic from the empirical statistic to generate an average bias, correct? To me that seems more like typical simulations. It makes sense, though. Now I'm going to try and see the world through bootstrapping lenses so I don't miss a chance to practice this!
    You should definitely use jQuery. It's really great and does all things.

  18. The Following User Says Thank You to bryangoodrich For This Useful Post:

    trinker (04-28-2015)

  19. #14
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: Feedback on bootstrapping effect size and t-values

    @BG Yes I believe so.

    Folks is it OK to bootstrap the effect size in this way as well?
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  20. #15
    Probably A Mammal
    Points: 31,087, Level: 100
    Level completed: 0%, Points required for next Level: 0
    bryangoodrich's Avatar
    Location
    Sacramento, California, United States
    Posts
    2,564
    Thanks
    398
    Thanked 618 Times in 551 Posts

    Re: Feedback on bootstrapping effect size and t-values


    I'd like an explanation of what you're trying to do instead of teasing it out of your code

    1. Handle the paired replication as Dason said
    2. Your computation looks fine
    3. Are you just taking the average effect size from the replicate effect sizes? This seems more like a simulation to me, whereas bootstrapping you would use the empirical ES (which the replicate one should be very close, but if you're doing a bias correction, hls' approach would apply here) and enclose it in the bootstrapped confidence interval. So is what you're after the CI or the ES measurement?
    You should definitely use jQuery. It's really great and does all things.

+ Reply to Thread
Page 1 of 3 1 2 3 LastLast

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats