+ Reply to Thread
Results 1 to 11 of 11

Thread: Should you use Bonferonni for regression

  1. #1
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Should you use Bonferonni for regression




    It is common to generate many models in regression and only report the final one. To me this suggests that you should use some correction like Bonferonni - but as best I can tell this is not done in regression. Similarly, it would seem you should use this when you test lots of variables at one time in any model.

    I was curious on other posters views on this.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  2. #2
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Should you use Bonferonni for regression

    Or if you feel the approach is aggregous (fishing for significant coefficients a middle ground may be using 0.01 instead of 0.05. All depends on the scenario.
    Stop cowardice, ban guns!

  3. #3
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Should you use Bonferonni for regression

    That seems reasonable. In my case it is not so much fishing as I have no idea what I am doing at first and have to build an understanding of the question at hand.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  4. #4
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Should you use Bonferonni for regression

    This isnt directly related but I was reading today on k-nearest neighbor, and it was talking about when you get above say three varables - hyper space gets pretty wonky and eventually you will find neighbors by chance but there are not what you were looking for.
    Stop cowardice, ban guns!

  5. #5
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Should you use Bonferonni for regression

    hi,
    I think Bonferroni is way too conservatve. Maybe you could look at this alternative: http://www.glicko.net/research/multiple-tests.pdf

    If you have a lot of data cross validation could also be a good criterion.

    regards

  6. The Following User Says Thank You to rogojel For This Useful Post:

    noetsi (02-01-2016)

  7. #6
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Should you use Bonferonni for regression

    Cross validation is a good idea. Also to control for overfitting. Unfortunately for me i typically have to have a priority hypotheses which means minimal sample size to find the minimal effect size.
    Stop cowardice, ban guns!

  8. #7
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Should you use Bonferonni for regression

    I don't know what cross validation is, but I will try to find out
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  9. #8
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Should you use Bonferonni for regression

    Where you train a model on a subsample of your data, then you test it on another (different) subsample of data, the test set. There is also, k-fold crossvalidation.
    Stop cowardice, ban guns!

  10. #9
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Should you use Bonferonni for regression

    You could simply forget the p-values and go for the model selection using the LASSO as discussed here, in a slightly different context:
    http://www.talkstats.com/showthread....el-of-a-factor

  11. The Following User Says Thank You to rogojel For This Useful Post:

    noetsi (02-01-2016)

  12. #10
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Should you use Bonferonni for regression

    Quote Originally Posted by hlsmith View Post
    Where you train a model on a subsample of your data, then you test it on another (different) subsample of data, the test set. There is also, k-fold crossvalidation.
    I just call this validation, I use it all the time with time series with a hold out data set. But I have not used it much with cross sectional data.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  13. #11
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Should you use Bonferonni for regression


    I never head of LASSO, but I will soon
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats