ROC significance

[Tikkuhirvio]

Hi all!

I'm using ROC analysis and would need to calculate whether a certain area under the ROC curve differs from 0.50 significantly. Here is what I think I should get out: http://www.cmh.edu/stats/ask/roc.asp (first table, "asymptotic significance"), but since I'm not primarily using SPSS (I'm using Matlab), I should find another way to get the significance. Which one of the references explains how to calculate the significance (I couldn't find the right one..)?

Also, is there any point in seeing if the 0.50 is included in the confidence interval (between the lower and higher confidence bound? Has anyone used this approach?

Thank you in advance.

Best Regards, Tikkuhirvio

[Link]

To get the area:
If you can fit an equation to the ROC curve, its as easy as integrating over the curve. From the reference you gave us, if you look at the first ROC curve graphed, it looks like there are just five lines connected together. So to get the area there, you would just do five integrations over the 5 lines.

Standard error
The reference you cited lists a standard error an CI for the ROC graph it lists. I'm tempted to tell you just to bootstrap the results and use the standard errors from it. However, doing a quick search, I came across a formula for you: http://www.anaesthetist.com/mnm/stats/roc/Findex.htm

Significance
I guess you could use a null hypothesis of having an area of 0.5. Since almost everytime, your curve will be above 0.5, you could be more liberal and use a one sided test (thought I'd recommend against it). You want to remember that the p-value and CI will be very similar in their results though. Almost everytime you see a p-value that is significant, you'll see a CI that crosses the null value.

[Tikkuhirvio]

Thank you, Link, for your reply.

I forgot to add one important detail to my post: The two distributions that I'm analyzing with the ROC curve are dependent, in fact, the patients are the same in both cases. Because of this, I believe the "Asymptotic significance" in the link that I posted cannot be used in my case. At least this is what I believe..

I can calculate the area under the ROC curve (AUC) with Matlab, and I can get the 95% confidence bounds to the AUC with bootstrapping (here I'm still a little unsure what the bootstrapping does to the dependent nature of the data sets..). Also, the link that you sent explains how to calculate the standard error of an AUC when the distributions are dependent. I'm not sure if I need the standard error, though.

Finally about the significance, I played with the SPSS a little bit and got a result where the 95% CI for the AUC included 0.5, but the p-value was still not significant (p>0.05). I think it may not be a good idea to rely on the CI.. But here is where the problem lies. The distribution are not normal and they are dependent, so if I would forget the ROC, I would most likely use something like the "Wilcoxon signed rank test" (is this ok?). This is what Matlab says about it:

"p = signrank(x,y) performs a paired, two-sided signed rank test of the null hypothesis that data in the vector x-y come from a continuous, symmetric distribution with zero median, against the alternative that the distribution does not have zero median."

But the null hypothesis is not that the true AUC is 0.5. What does this mean? How do I get such test done?

Thank you,
Tikkuhirvio