+ Reply to Thread
Results 1 to 5 of 5

Thread: ROC curve cutoff not significant, but it works

  1. #1
    Points: 5,312, Level: 46
    Level completed: 81%, Points required for next Level: 38

    Posts
    43
    Thanks
    28
    Thanked 1 Time in 1 Post

    ROC curve cutoff not significant, but it works




    I'm dealing with a numerical variable and its prognostic significance in a sample of cancer patients. I'm trying to find an optimal cutoff point for survival analysis and therefore I used ROC curve analysis with survival outcome as classification variable (dead/alive) although I know this is not perfect solution but it is widely used in medical literature.

    But here is the problem: statistical program offers me optimal cutoff point together with AUC that turns out to be 0.608 (a bit low), and p=0.08 (not significant).
    This cutoff point however offers very significant result when used for log-rank test and later in Cox regression analysis. Even more, dichotomous status regarding elevation in mentioned parameter turns out to predict survival independently of current prognostic staging system (both very significant results) and other factors affecting survival. Since this offers an evidence that parameter bears very good additional prognostic value, does this allow an use of previously insignificant (borderline significant) result? How should I explain this fact in a manuscript? What would be the conclusion of ROC curve analysis?

  2. #2
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: ROC curve cutoff not significant, but it works

    Did you control for time to event? I believe there is a version of ROC Curve that you calculate it with survival model.
    Stop cowardice, ban guns!

  3. #3
    Points: 5,312, Level: 46
    Level completed: 81%, Points required for next Level: 38

    Posts
    43
    Thanks
    28
    Thanked 1 Time in 1 Post

    Re: ROC curve cutoff not significant, but it works

    No, I know this is a potential issue but I'm not aware of this option (I'm searching for it for a long time) and what statistical program could do it. Thank You for bringing this up.

    Since in this case described here ROC curve analysis is just a tool to find a cutoff with maximal effect on survival (that will be used with another test) and not the end of the road, somehow I feel this is not a very big issue. However, I'm not sure how to interpret these ROC curve analysis results and what conclusions can I draw from it. Since censoring is ignored, ROC curve analysis can not be sole mean to determine that this variable has good predictive value in terms of mortality (I guess that would be the case if the test was significant too).
    Last edited by Markica85; 09-04-2016 at 07:36 AM.

  4. #4
    Points: 228, Level: 4
    Level completed: 56%, Points required for next Level: 22

    Location
    Tampa, FL
    Posts
    24
    Thanks
    0
    Thanked 11 Times in 10 Posts

    Re: ROC curve cutoff not significant, but it works

    What you just described is actually quite common; the AUC test statistic is known to be conservative, and so the test lacks power, not to mention the potential power loss from transforming the survival time outcome to a dichotomous one. See:

    Seshan, V. E., Gönen, M., & Begg, C. B. (2013). Comparing ROC Curves Derived From Regression Models. Statistics in Medicine, 32(9), 1483–1493. http://doi.org/10.1002/sim.5648

    Also, let's remember that p=0.05 is by no means the holy threshold of scientific truth—particularly for observational data, where significance depends on whatever sample size you have available.
    http://ab-stats.com

  5. The Following User Says Thank You to ab-stats For This Useful Post:

    Markica85 (09-05-2016)

  6. #5
    Omega Contributor
    Points: 38,289, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,992
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: ROC curve cutoff not significant, but it works


    I was thinking of the SAS Gonen book.
    Stop cowardice, ban guns!

  7. The Following User Says Thank You to hlsmith For This Useful Post:

    Markica85 (09-05-2016)

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats