+ Reply to Thread
Results 1 to 10 of 10

Thread: Multiple imputation and ROC curve anlaysis

  1. #1
    Points: 3,015, Level: 33
    Level completed: 77%, Points required for next Level: 35

    Posts
    47
    Thanks
    8
    Thanked 7 Times in 7 Posts

    Multiple imputation and ROC curve anlaysis




    Hi,
    I have searched extensively for a solution to the following problem, without success.
    I have used multiple imputation on SPSS 22 to create 5 datasets which have a number of variables that predict patient outcome.

    As mentioned in a number of previous threads, SPSS does not pool the analysis for certain types of tests, though these can be done by hand using Rubin’s method: http://sites.stat.psu.edu/~jls/mifaq.html#howto

    I have used this approach (checked and re-checked multiple times, and checked the original Rubin texts) and my result seems incorrect. In short, the overall standard error is far higher than in any of the 5 datasets or the original dataset. When I used the same pooled analysis to compare one ROC curve to another ROC curve (pooling the standard error for the differences in the areas of the ROC curves) the result is far from significant. However, in the original dataset and the 5 imputed sets, the p value for the comparison is highly significant: <0.0001.

    I would have assumed that an analysis using imputed data would not result in far wider 95% CI and a higher p value then a complete case analysis (the original dataset)?!

    Is Rubin’s approach not accurate for ROC curves (though the literature is full of people “pooling” ROC curves citing his approach) or have I made a simple error?!

    Any help is greatly appreciated.

  2. #2
    Omega Contributor
    Points: 38,396, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,001
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Not my area, but how much missing data are we talking?
    Stop cowardice, ban guns!

  3. The Following User Says Thank You to hlsmith For This Useful Post:

    CE479 (04-01-2015)

  4. #3
    Points: 3,015, Level: 33
    Level completed: 77%, Points required for next Level: 35

    Posts
    47
    Thanks
    8
    Thanked 7 Times in 7 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Missing data for two variables is 12.2% and 11.7%; the rest (13 variables) are less than 3.5%. I am reasonably confident that that MI is the correct approach, and that 5 datasets are sufficient (though welcome comments about this aspect too). It is the pooling of ROC curves that is making me scratch my head!

  5. #4
    Points: 3,882, Level: 39
    Level completed: 55%, Points required for next Level: 68

    Posts
    47
    Thanks
    2
    Thanked 13 Times in 13 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Quote Originally Posted by CE479 View Post
    Missing data for two variables is 12.2% and 11.7%; the rest (13 variables) are less than 3.5%. I am reasonably confident that that MI is the correct approach, and that 5 datasets are sufficient (though welcome comments about this aspect too). It is the pooling of ROC curves that is making me scratch my head!
    Are you trying to create a pooled ROC curve (graph) from the 5 imputed data sets? I have not seen that before. Or are you trying to evaluate the pooled AUC estimate? Or is it the difference between AUC estimates? I believe the CI should be a bit wider due to the uncertainty inherent in the imputation process, but if you are getting p<.0001 in the complete case analysis and each imputed data set, something does not seem right. I would check your math again. Make sure you are pooling the correct standard error, too, for whatever evaluation you are doing. If it is the difference between AUC estimates, I believe you need the standard error of the AUC difference for each imputed data set.

  6. #5
    Points: 3,015, Level: 33
    Level completed: 77%, Points required for next Level: 35

    Posts
    47
    Thanks
    8
    Thanked 7 Times in 7 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Thanks for replying, Mostater.
    There are two things that I am trying to do:
    1) Create a pooled AUROC and 95% CI from the 5 datasets (+/- the ROC curve graph.)
    2) Compare the AUROC between one score, and on a score that has had data imputed. As you say, for this you need to compare the standard error of the AUC difference for each imputed dataset, which I have tried to do. I may have made a mistake, but have checked numerous times and cannot find one- see below.

    I was wondering if the problem was related to the fact that ROC curve areas are between 0-1, and whether I needed to transform data first?

    I worked through Rubins method: http://sites.stat.psu.edu/~jls/mifaq.html#howto

    Here are the area differences between the scores for each imputed dataset, and the corresponding standard error:
    ScoreA v ScoreB_1 0.05047 0.02443
    ScoreA v ScoreB_2 0.04972 0.02441
    ScoreA v ScoreB_3 0.0529 0.02476
    ScoreA v ScoreB_4 0.05307 0.02476
    ScoreA v ScoreB_5 0.0486 0.02414

    This pooled area difference is: 0.050952 (Sum of first column/ 5)

    The within imputation variance is: 0.0245 (Sum of second column/ 5)

    The between imputation variance is: 3.89067E-06 (the differences between each area difference and the pooled area difference squared, then added up, then divided by 4)

    The total variance is: 0.024504669 (The within imputation variance plus 1.2*between variance)

    The overall standard error is: 0.156539672 (square root of total variance)

    Thanks for any help!

  7. #6
    Points: 3,882, Level: 39
    Level completed: 55%, Points required for next Level: 68

    Posts
    47
    Thanks
    2
    Thanked 13 Times in 13 Posts

    Re: Multiple imputation and ROC curve anlaysis

    I followed the PSU link and it appears that you are correctly applying your #s to the formulas presented. However, when I compare the PSU algorithm for combining results to those in SAS, the instructions are the same, but SAS requires the within-imputation variance as opposed to the standard error. So my thought is that you need to get to the estimated error variance by squaring the standard error first, then do the calculations.

    Here is a link to the SAS documentation ... http://support.sas.com/documentation...ze_sect012.htm

  8. The Following User Says Thank You to mostater For This Useful Post:

    CE479 (04-02-2015)

  9. #7
    Points: 3,015, Level: 33
    Level completed: 77%, Points required for next Level: 35

    Posts
    47
    Thanks
    8
    Thanked 7 Times in 7 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Thanks mostater- I will have a look at this and hopefully return with the correct answer!

  10. #8
    Points: 3,015, Level: 33
    Level completed: 77%, Points required for next Level: 35

    Posts
    47
    Thanks
    8
    Thanked 7 Times in 7 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Thanks again, Mostater- this approach appears to be effective.

    I have had a look at this and re-run the calculation squaring the standard error.

    This gives an overall standard error of 0.0246 which is on the high side of the standard errors from the individual datasets- in other words, what I would expect!

    My only query is regards converting the standard error into the variance- when I have done such calculations by hand in the past, usually I have taken the sample size into account (i.e. SE= SD/ n^1/2). In this instance is n effectively 1 because it is a single dataset? Is that a silly question?!

    I have gotten the answer that I want, but I just want to be 100% about the approach!
    Last edited by CE479; 04-04-2015 at 06:21 AM.

  11. #9
    Points: 3,882, Level: 39
    Level completed: 55%, Points required for next Level: 68

    Posts
    47
    Thanks
    2
    Thanked 13 Times in 13 Posts

    Re: Multiple imputation and ROC curve anlaysis

    Think of the standard error as the standard deviation of the estimate sampling distribution. And just like when you square the standard deviation and get the variance, you can square the standard error to get the error variance of the estimate.

    Using the mean as an example, the standard error of the mean is the standard deviation of the sampling distribution of the mean (imagine taking many, many samples all of the same size, obtaining the mean for each sample, and then examining the histogram of all those means...that's the sampling distribution of the mean). It can be estimated by taking the standard deviation of the underlying data and dividing by sqrt(n). And if you wanted the error variance of the mean, you would just square the standard error. The focus is typically on the standard error though. Most tests and confidence intervals are based on that vs. the error variance.

    It can get confusing and the jargon sometimes doesn't help, but I hope my explanation does.

  12. The Following User Says Thank You to mostater For This Useful Post:

    CE479 (04-02-2015)

  13. #10
    Points: 3,015, Level: 33
    Level completed: 77%, Points required for next Level: 35

    Posts
    47
    Thanks
    8
    Thanked 7 Times in 7 Posts

    Re: Multiple imputation and ROC curve anlaysis


    That makes perfect sense, thanks to your clear explanation.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats