+ Reply to Thread
Results 1 to 9 of 9

Thread: Logistic regression convergence issues

  1. #1
    Points: 2,626, Level: 31
    Level completed: 18%, Points required for next Level: 124

    Location
    Dallas, TX
    Posts
    311
    Thanks
    12
    Thanked 90 Times in 88 Posts

    Logistic regression convergence issues




    Hi Folks!

    I'm using logistic regression for classifying records into groups. The issue with logistic regression is that if any cell in the contigency table of dependent v/s independent equals 0 then there are convergence issues. Usually the program warns with messages like "complete separation/ quasi- complete separation" etc. and there are no unique maximum likelihood solutions in this case.

    To explain what I mean by contingency table of DV v/s IV, if the equation is Logit(Y)=a+X1+X2+... then frequency table of Y v/s X1, Y v/s X2 and so on. Say if any of the cells of Y v/s X1 is 0 then algorithm won't converge. If I remove X1 from analysis then it converges but unfortunately, I can't remove these variables.

    My questions are as follows:
    1) Does anyone have any idea if Discriminant analysis also suffers from this problem?

    2) Any alternatives to solving this problem other than logistic regression and discriminant analysis? The solution should be well implemented by the softwares.

    3) Even if there is no unique solution, the program still produces estimates based on the last iteration. In this case, are odds ratios or even parameter estimates robust? By robust I mean can I safely draw conclusions on them expecting that the unique solution would be approximately close to this.

    Your positive response is deeply appreciated.

  2. #2
    Point Mass at Zero
    Points: 13,828, Level: 76
    Level completed: 45%, Points required for next Level: 222
    ledzep's Avatar
    Location
    Berks,UK
    Posts
    684
    Thanks
    188
    Thanked 143 Times in 139 Posts

    Re: Logistic regression convergence issues

    Sorry, it might have been a bit late answer. Although I am not intending to answer your 3 questions specifically, I want to share my thoughts.
    I have suffered similar problems in the past when there were 0 counts in some of the groups, which resulted in massive odds ratio and large standard errors. I got around the problem by combing some of the levels of the factor so that each level of factor had a non zero count.
    Oh Thou Perelman! Poincare's was for you and Riemann's is for me.

  3. #3
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Logistic regression convergence issues

    Gelman has a paper (LINK) on logistic regression with weakly informative priors, which will purportedly provide sensible parameter estimates even in the face of complete separation. The paper includes example code for fitting such models using the bayesglm() function from the "arm" package in R. I haven't worked through the paper or played with the function, so I can't say much beyond this.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  4. The Following User Says Thank You to Jake For This Useful Post:

    jrai (04-04-2012)

  5. #4
    Devorador de queso
    Points: 95,540, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,930
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Logistic regression convergence issues

    Yay! Bayesian analysis FTW.
    I don't have emotions and sometimes that makes me very sad.

  6. #5
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Logistic regression convergence issues

    I thought that might make you happy.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  7. #6
    Devorador de queso
    Points: 95,540, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,930
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Logistic regression convergence issues

    To be honest my first thought when I saw the OP was a Bayesian approach. I probably wouldn't have taken exactly the same approach as Gelman but I would trust Gelman's advice more than mine on this type of issue. But I believe jrai is mainly a SAS person and I don't know if SAS has that capability so I skuttled away from that option for a while. Although it does have proc MCMC so you could program it yourself...
    I don't have emotions and sometimes that makes me very sad.

  8. The Following User Says Thank You to Dason For This Useful Post:

    jrai (04-04-2012)

  9. #7
    Point Mass at Zero
    Points: 13,828, Level: 76
    Level completed: 45%, Points required for next Level: 222
    ledzep's Avatar
    Location
    Berks,UK
    Posts
    684
    Thanks
    188
    Thanked 143 Times in 139 Posts

    Re: Logistic regression convergence issues

    SAS seem to have the Bayesian version of most of the PROCs.

    For eg: PROC BGENMOD, PROC BLIFEREG, PROC BPHREG.
    http://support.sas.com/rnd/app/da/bay******.html
    Oh Thou Perelman! Poincare's was for you and Riemann's is for me.

  10. The Following User Says Thank You to ledzep For This Useful Post:

    jrai (04-04-2012)

  11. #8
    Devorador de queso
    Points: 95,540, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,930
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Logistic regression convergence issues

    Well look at that. I guess I don't pay too much attention to SAS news.
    I don't have emotions and sometimes that makes me very sad.

  12. #9
    Points: 2,626, Level: 31
    Level completed: 18%, Points required for next Level: 124

    Location
    Dallas, TX
    Posts
    311
    Thanks
    12
    Thanked 90 Times in 88 Posts

    Re: Logistic regression convergence issues


    Quote Originally Posted by ledzep View Post
    Sorry, it might have been a bit late answer. Although I am not intending to answer your 3 questions specifically, I want to share my thoughts.
    I have suffered similar problems in the past when there were 0 counts in some of the groups, which resulted in massive odds ratio and large standard errors. I got around the problem by combing some of the levels of the factor so that each level of factor had a non zero count.
    Thanks for the idea. I thought about this but it doesn't work for the client though. Well I found that the results for variables with non-zero counts are robust. I compared the results from model with all the variables with those from the model with variables with non-zero count and interestingly the results are very close. I don't know whether this is generally true or not but for the current case it seems to be true.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats