# Thread: ROC curve cutoff not significant, but it works

1. ## ROC curve cutoff not significant, but it works

I'm dealing with a numerical variable and its prognostic significance in a sample of cancer patients. I'm trying to find an optimal cutoff point for survival analysis and therefore I used ROC curve analysis with survival outcome as classification variable (dead/alive) although I know this is not perfect solution but it is widely used in medical literature.

But here is the problem: statistical program offers me optimal cutoff point together with AUC that turns out to be 0.608 (a bit low), and p=0.08 (not significant).
This cutoff point however offers very significant result when used for log-rank test and later in Cox regression analysis. Even more, dichotomous status regarding elevation in mentioned parameter turns out to predict survival independently of current prognostic staging system (both very significant results) and other factors affecting survival. Since this offers an evidence that parameter bears very good additional prognostic value, does this allow an use of previously insignificant (borderline significant) result? How should I explain this fact in a manuscript? What would be the conclusion of ROC curve analysis?

2. ## Re: ROC curve cutoff not significant, but it works

Did you control for time to event? I believe there is a version of ROC Curve that you calculate it with survival model.

3. ## Re: ROC curve cutoff not significant, but it works

No, I know this is a potential issue but I'm not aware of this option (I'm searching for it for a long time) and what statistical program could do it. Thank You for bringing this up.

Since in this case described here ROC curve analysis is just a tool to find a cutoff with maximal effect on survival (that will be used with another test) and not the end of the road, somehow I feel this is not a very big issue. However, I'm not sure how to interpret these ROC curve analysis results and what conclusions can I draw from it. Since censoring is ignored, ROC curve analysis can not be sole mean to determine that this variable has good predictive value in terms of mortality (I guess that would be the case if the test was significant too).

4. ## Re: ROC curve cutoff not significant, but it works

What you just described is actually quite common; the AUC test statistic is known to be conservative, and so the test lacks power, not to mention the potential power loss from transforming the survival time outcome to a dichotomous one. See:

Seshan, V. E., Gönen, M., & Begg, C. B. (2013). Comparing ROC Curves Derived From Regression Models. Statistics in Medicine, 32(9), 1483–1493. http://doi.org/10.1002/sim.5648

Also, let's remember that p=0.05 is by no means the holy threshold of scientific truth—particularly for observational data, where significance depends on whatever sample size you have available.

5. ## The Following User Says Thank You to ab-stats For This Useful Post:

Markica85 (09-05-2016)

6. ## Re: ROC curve cutoff not significant, but it works

I was thinking of the SAS Gonen book.

7. ## The Following User Says Thank You to hlsmith For This Useful Post:

Markica85 (09-05-2016)

 Tweet

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts