+ Reply to Thread
Results 1 to 6 of 6

Thread: Bootstrap procedure to test difference between correlation coefficients

  1. #1
    Points: 156, Level: 3
    Level completed: 12%, Points required for next Level: 44

    Location
    Italy
    Posts
    5
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Bootstrap procedure to test difference between correlation coefficients




    Hi everyone,

    I have two correlation coefficients (r1 and r2), obtained within the same sample (20 subjects). My aim is to test it they are significantly different.
    r1 is the correlation between a neurophysiological parameter and a behavioural parameter in condition A; r2 is the correlation between the same neurophysiological parameter and the same behavioural parameter in condition B.

    I was thinking to apply a bootstrap procedure for each condition, in order to obtain two distributions of correlations. Then, I can simply run a two-sample t-test, to test for a significant difference.
    My questions:
    1) Does this procedure seem reasonable to achieve my purpose? (test if r1 and r2 are significantly different);
    2) There is a way to decide the number of iterations, or is it totally arbitrary? (for example..I can go with 1000?).

    Thanks in advance,
    Simone

  2. #2
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Bootstrap procedure to test difference between correlation coefficients

    Hmm, I am not an expert in this area but this seems off. Your general intentions are fine, though I don't think you need to run a t-test. Can you describe the groups and variables in more detail? It seems like you are saying they are the same variables and people, so I am not seeing the distinction between the two r's. And why can you use the same sample to calculate both parameters?


    A thousand resamples is usually fine. You can also run say 500, 1000, 5000, and 10000, then see if they all converged to about the same distribution (that is the rule of thumb). If you think about the resampling and then doing a t-test, its outcome can change based on the number of resamples. So you can decrease your standard errors more and more just be taking more resamples. That doesn't seem right to use a your deciding factor.


    What is the explicit question you are trying to answer and what is your hypothesis about that question?
    Stop cowardice, ban guns!

  3. The Following User Says Thank You to hlsmith For This Useful Post:

    smndpln (11-16-2016)

  4. #3
    Points: 156, Level: 3
    Level completed: 12%, Points required for next Level: 44

    Location
    Italy
    Posts
    5
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: Bootstrap procedure to test difference between correlation coefficients

    Hi hlsmith, and thanks for the reply.

    Actually, the two r's are different because they come from different condition in an experimental paradigm. So I have r1, which is the correlation between reaction times in the negative condition and brain connectivity in the negative condition, and r2, which is the correlation between reaction times in the neutral condition and brain connectivity in the neutral condition.

    The explicit question is: are the two correlations different? Hypotesis: yes, at least for the connectivity in specific brain regions.

    P.S. Probably, you're going to think this is a job for an ANOCOVA with separate regression slopes. Anyway, I noticed that ANCOVA is likely to accept null hypotesis with small sample sizes. Also Cohen's d may be useful, but does not give a statistical significance.

  5. #4
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Bootstrap procedure to test difference between correlation coefficients

    I believe you can do this, however using the following stipulations:


    -Sent your seed for the bootstrap resamples and run both correlations on the same data sets.


    -Find the differences between the two calculated r's, so match them based on the same resamples (pairs) and subtract.


    -Run a paired t-test.


    Though, as I mentioned earlier the number of resamples is unbound. So, I guess you end up powering to the level you are comfortable with. So you could do 50 our 10,000, which the later would end-up being an pretty steep distribution, but could get you close to the truth, you may try not to report the p-value, but an effect size. This would also benefit from graphing a final histogram of the correlations as well as their differences.
    Stop cowardice, ban guns!

  6. #5
    Points: 1,974, Level: 26
    Level completed: 74%, Points required for next Level: 26

    Location
    New Zealand
    Posts
    227
    Thanks
    3
    Thanked 48 Times in 47 Posts

    Re: Bootstrap procedure to test difference between correlation coefficients

    You probably want to keep the fact that the data comes from the same subjects. Here is a suggestion.
    Put the data in columns with the subject as an index column. Find the two correlations and find the difference.
    Resample the rows with replacement. This is really just resampling the subjects. Record the new difference. Do this 1000 times, or 2000 it only makes 10 seconds difference. Find the percentile that ZERO is. If this is over 0.5 then take it away from 1. This is a one sided p value. Doubler it for your two sided value.
    Last edited by katxt; 11-19-2016 at 02:15 PM. Reason: Correct crucial error

  7. #6
    Points: 156, Level: 3
    Level completed: 12%, Points required for next Level: 44

    Location
    Italy
    Posts
    5
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: Bootstrap procedure to test difference between correlation coefficients


    hlsmith, I think you are right and maybe the estimation of effect size is better. I'm using the classic Cohen' d for this.

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats