+ Reply to Thread
Results 1 to 7 of 7

Thread: Critiques Requested on Resampling as an Approach

  1. #1
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Critiques Requested on Resampling as an Approach




    I found an old piece code I wrote. I had a logistic model with a multiplicative interaction term consisting of two binary variables plus model covariates. For simplicity, I will say the sample size was 200. The interaction term was approaching significance and made contextual sense.


    I ran a bootstrap of a few thousand resamples and made the sample rate 300% or 600 observations instead of the original 200. I then reran the model with the 3000 samples and graphed out all generated interaction term p-values. Next, I looked to see if the Bootstrap 95% CI excluded values > 0.05. I did all of this as a pseudo sample size calculation. I will mention that the created p-value distribution was not no normally distributed. It was very positively skewed, with most values near 0 and a very few trickling down toward 1. Obviously, it was bounded. The mean of the distribution was 0.07 and median 0.01.


    I believe my rationale for this approach was that I had difficulties with calculating the required sample size given more traditional approaches based on the multiple logistic model. What are critiques of this approach and alternatives?

  2. #2
    Devorador de queso
    Points: 95,540, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,930
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Critiques Requested on Resampling as an Approach

    What exactly was your goal? And we would pretty much never expect a p-value distribution to be normally distributed.
    I don't have emotions and sometimes that makes me very sad.

  3. #3
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Critiques Requested on Resampling as an Approach

    The goal was to see if the sample size was three times larger, whether the SE would shrink up and the same effect would be statistically significant. Based on the SE should reduce in the presence of a larger sample regardless of the approximate stagnant effect size.


    I know the approach is flawed in some regard, but I need some one to point out the issues.

    Some variables are not symmetric under the sampling distribution. Are p-values an issue since they are derived from another distribution and/or bounded?

  4. #4
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Critiques Requested on Resampling as an Approach

    Ah, the p-value is a random variable (which is conditional on the null hypothesis), which means it has a sampling distribution. I read that when the null is true it will take on a uniform distribution, however when the alternative is true it takes on a positively skewed distribution. Like mine, see below. So what issues may arise from my approach in general? I am wondering about the sampling rate (e.g., 300) and using 95% on a non-normal distribution?
    Attached Images  

  5. #5
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Critiques Requested on Resampling as an Approach

    For fun, I log transformed the above distribution and got close to normal. Though, as expected that did not change my 2.5% and 97.5 estimates (when transforming back using e).

  6. #6
    Devorador de queso
    Points: 95,540, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,930
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Critiques Requested on Resampling as an Approach

    I'm not entirely sure why you're worrying about quantiles here. Typically if you generate the p-value distribution for this reason you look at the estimated power. So you would want to look at what percentage of those p-values fall below your cutoff (0.05).
    I don't have emotions and sometimes that makes me very sad.

  7. #7
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Critiques Requested on Resampling as an Approach


    EDITED: You told me this once before, 1.5 years ago - perhaps.


    So I wrapped my head around this. Would I be concerned about Type I Error, and how do you address that?

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats