+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 20

Thread: how to calculate the significance of an R2 increase (based only on results in paper)

  1. #1
    Points: 573, Level: 11
    Level completed: 46%, Points required for next Level: 27

    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    how to calculate the significance of an R2 increase (based only on results in paper)




    Recently I have come across several articles that report OLS results of models with increasing numbers of variables. In the articles the R2 and F-statistics are reported (and, of course, t or p values for the coefficients).
    In these articles, I note that the R2 (or adjusted R2) doesn't increase very much and expect that the increase in R2 (or adjusted R2) may not be significant. However, the authors do not provide the F-test of the significance of the increase. Therefore, I would like to calculate this by hand. Can anyone tell me what the formula for this is?

    For example, one study has two models with adjR2=.125 and .168. These two models vary by only 1 variable (which is included in model 2 but not in model 1). The accompanying F-stats are 3.113 and 4.848, the number of coefficients are 6 and 7 respectively, the number of observations 140.
    The aythors claim that this additional variable is very important (it is stat.sign), but I wonder whether the increase in adjR2 is high enough to even continue with the larger model. How would I calculate that based on the reported results table only? I don't have the actual data, so can only work with what is reported in the article.
    If possible, I'd like to know how to do this both for R2 and for adjR2, given that some authors report R2 and others report adjR2.

    Thanks, Peter

  2. #2
    Points: 547, Level: 10
    Level completed: 94%, Points required for next Level: 3

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    I don't know how to do this with R2 but if you have t and p-values for the coefficients, than all variables with p>0.05 are insignificant (if p-values are calculated correctly). Generally, adjR2 are too small for the applicable model. I can say exactly due to my experience - 1. there are redundant variables in this model 2. This model is garbage even if it is statistically significant ( - not applicable). Tip to the author of article - identify the most significant predictor and say that there is a weak correlation. This will be honestly. Other is pseudoscience.

  3. #3
    Points: 610, Level: 12
    Level completed: 20%, Points required for next Level: 40

    Posts
    11
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Correct me if I'm wrong but I don't think you can compare 2 R squared results and say that one is statistically significant and the other not because they are a measure of effect size rather than statistical significance. You would assess statistical significance on the .05 P value criterion and if the F test met the appropiate critical value in the F distribution.

  4. #4
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Quote Originally Posted by PeterV View Post
    Recently I have come across several articles that report OLS results of models with increasing numbers of variables. In the articles the R2 and F-statistics are reported (and, of course, t or p values for the coefficients).
    In these articles, I note that the R2 (or adjusted R2) doesn't increase very much and expect that the increase in R2 (or adjusted R2) may not be significant. However, the authors do not provide the F-test of the significance of the increase. Therefore, I would like to calculate this by hand. Can anyone tell me what the formula for this is?

    Thanks, Peter

    What the authors are reporting is all that is needed. That is, additional information would be redundant.

    You only need to know the t and p-values associated with the regression weights. That's the information that is addressing the question you're asking.

    A t statistic (associated with any of the regression weights) is testing the unique contribution of a specific independent variable i.e. as if it were entered last into the model.

  5. #5
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Quote Originally Posted by Ventures
    Correct me if I'm wrong but I don't think you can compare 2 R squared results and say that one is statistically significant and the other not because they are a measure of effect size rather than statistical significance.
    The article "Correlation Redux" by Finn and Oklin (spelled wrong) attempts to compare 2 . Cohen, cohen Aiken and West recomend this procedure as outlined by Finn and Oklin. The procedure is recomended for larger samples though. But in this case we're dealing with which such procedures wouldn't really be appropriate for as they are strength of effect measures (as you pointed out).
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  6. #6
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    You can do a F change test to determine if new predictors added signficant additional predictive power. In this case variables are added hierarchially rather than all at once. But I think you would have to have the raw data to do this.

  7. #7
    Points: 573, Level: 11
    Level completed: 46%, Points required for next Level: 27

    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Quote Originally Posted by Dragan View Post
    What the authors are reporting is all that is needed. That is, additional information would be redundant.

    You only need to know the t and p-values associated with the regression weights. That's the information that is addressing the question you're asking.

    A t statistic (associated with any of the regression weights) is testing the unique contribution of a specific independent variable i.e. as if it were entered last into the model.
    I see that. Maybe my point wasn't entirely clear. When adding one or more variables to a regression mode, these new variables may or may not be significant. At the same time, the coefficients and significance of the "previous" variables are affected.
    Even when the newly added variable reaches statistical significance, it is not necessarily the case that that the larger model as a whole is preferable. Although R2 increases, the increase in R2 (as an indicator of the fit of the whole model) can be so small that the more parsimonious model is to be preferred.
    As far as I know, it is common to calculate an F-statistic to test whether the fit of the larger model is stat. sign better than that of the smaller model. Basically, this is an AOV. I know how to calculate this in standard statistical software, when I have the data available. But my quest of for how to calculate this F-statistic (comparing two models with one nested in the other) when I only have the regression results available.

  8. #8
    Points: 573, Level: 11
    Level completed: 46%, Points required for next Level: 27

    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Quote Originally Posted by noetsi View Post
    You can do a F change test to determine if new predictors added signficant additional predictive power. In this case variables are added hierarchially rather than all at once. But I think you would have to have the raw data to do this.
    this is exactly what I am looking for, except for the "I think you would have to have the raw data to do this" part :-)
    Would there be no way to do this based on the info reported in the regression results in an article?

  9. #9
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    None that I am familar with. You could look up F change test and aggregate data and see if there is a way, but I doubt there is. Perhaps a metaanalysis technique could be adapted to address, but again I doubt that very much.

  10. #10
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    This is likely covered by a meta analysis technique. As noetsi points out this problem would be easy to conduct with raw data but with a few stats from the article your job becomes harder. I personally don't know the answer to your question but here is a link about comparing models that may be of use (LINK). It appears to give formulas to do what you want. Sum of squares can be figured out from the anova table [usually] provided in the article from df and the f stat.

    EDIT: funny both noetsi and I recommended a meta analysis technique at the same time
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

  11. #11
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    You could also contact the author(s) and ask them for the raw data. I have gotten data files from authors to conduct research (they usually are happy to give it to you).

  12. #12
    Ninja say what!?!
    Points: 8,297, Level: 61
    Level completed: 49%, Points required for next Level: 153
    Link's Avatar
    Posts
    1,165
    Thanks
    37
    Thanked 84 Times in 76 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Quote Originally Posted by noetsi View Post
    You could also contact the author(s) and ask them for the raw data. I have gotten data files from authors to conduct research (they usually are happy to give it to you).
    Wow. Really noetsi?!?!? Every researcher I've come across has been overly safe and frugal when it's come to data sharing.

  13. #13
    TS Contributor
    Points: 22,411, Level: 93
    Level completed: 7%, Points required for next Level: 939
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    well... i guess i am gonna have to assume i am the only one who has ever been exposed to this equation but here we go. to test whether the increase in the R^{2} when adding another variable (or sets of variables like in hierarchical regression) into the model is significant we use the following F-statistic:

    F = \frac{R_{full}^{2}-R_{reduced}^{2}}{1-R_{full}^{2}}\cdot \frac{n-p_{full}-p_{reduced}-1}{p_{full}}

    Where:

    R_{full}^{2} is the R^{2} of the regression equation with the LARGERnumber of predictors

    R_{reduced}^{2} is the R^{2} of the regression equation with the SMALLER number of predictors (which is nested in the larger model)

    n is your sample size

    p_{full} is the number of predictors in your full model (also the degrees of freedom for your numerator in the F-ratio)

    p_{reduced} is the number of predictors in your smaller model

    and n-p_{full}-p_{reduced}-1 is the degrees of freedom in the denominator of your F-ratio, for when you go and check for significance.

    this is the F-test that any software does to test whether the increment in R^{2} is statistically significant or not.

    from what i read on your original post, you should be able to calculate the F statistic and check its significance with the data at hand.

    have fun!
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  14. #14
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap

    Quote Originally Posted by spunky View Post
    well... i guess i am gonna have to assume i am the only one who has ever been exposed to this equation but here we go. to test whether the increase in the R^{2} when adding another variable (or sets of variables like in hierarchical regression) into the model is significant we use the following F-statistic:

    F = \frac{R_{full}^{2}-R_{reduced}^{2}}{1-R_{full}^{2}}\cdot \frac{n-p_{full}-p_{reduced}-1}{p_{full}}
    Ahem, check your degrees of freedom in the numerator and denominator, Spunky.

  15. #15
    ggplot2orBust
    Points: 71,220, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    User with most referrers
    trinker's Avatar
    Location
    Buffalo, NY
    Posts
    4,417
    Thanks
    1,811
    Thanked 928 Times in 809 Posts

    Re: how to calculate the significance of an R2 increase (based only on results in pap


    Oh yeah I had to calculate that in a linear regression class it's in Cohen Cohen Aiken and West. duh. That's a pretty common formula. Thanks for making me feel like a real ding a ling spunky If my regression prof were to see this he'd not be happy.
    "If you torture the data long enough it will eventually confess."
    -Ronald Harry Coase -

+ Reply to Thread
Page 1 of 2 1 2 LastLast

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats