+ Reply to Thread
Results 1 to 11 of 11

Thread: cross-validated AUC

  1. #1
    TS Contributor
    Points: 40,771, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Downloads
    gianmarco's Avatar
    Location
    Italy
    Posts
    1,370
    Thanks
    233
    Thanked 302 Times in 226 Posts

    cross-validated AUC




    Hello All,
    I hope the thread's title makes sense to you. I need to perform internal Cross-Validation using k-fold CV (needless to say, to assess how well a model behaves in relation to 'unknown' data).

    What I am after is getting the distribution of AUC values across the different folds. So far, I did not found a viable option. I mean, there are some packages that perform different sorts of CV, but no one of them (at the best of my understanding) return what I want.

    One that I found quite easy to use if the DAAG package, whose CVbinary() function performs k-folds CV and returns the cross-validation estimate of accuracy. The latter, as far as I understand, is the average of the accuracy across the k-folds (using 0.5 as cutoff point on probabilities).

    What I would like to have is something similar, but with the averaged AUCs instead of the averaged accuracy values.

    Long story short: do you know of any package that does something like that, or can you provide some help in writing down some piece of code to help me implementing what I am after from scratch?

    Thank you for any guidance you will provide.

    Best
    Gm
    http://cainarchaeology.weebly.com/

  2. #2
    Super Moderator
    Points: 31,766, Level: 100
    Level completed: 0%, Points required for next Level: 0
    bugman's Avatar
    Posts
    2,255
    Thanks
    290
    Thanked 324 Times in 265 Posts

    Re: cross-validated AUC

    Have you tried the function:

    Code: 
    cv.glm()
    in the boot package.

    It does k fold cross validation and may have some of the arguments that you require.

    Failing that, have you tried the package "cvAUC"? (I have not used that one).
    The earth is round: P<0.05

  3. #3
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: cross-validated AUC

    AUC = Accuracy = Discrimination?

  4. #4
    TS Contributor
    Points: 40,771, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Downloads
    gianmarco's Avatar
    Location
    Italy
    Posts
    1,370
    Thanks
    233
    Thanked 302 Times in 226 Posts

    Re: cross-validated AUC

    Hello!
    @Bugman:
    thanks for pointing out cv.glm from boot package. I was wondering what is the interpretation of the returned delta values.
    As for the cvAUC, I did not manage to have it work properly: I can't get the AUC for the various (say, 10) folds. I keep getting the AUC for just one fold :-(

    @hlsmith
    I did not get your question? Sorry
    http://cainarchaeology.weebly.com/

  5. #5
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: cross-validated AUC

    You said averaged accuracy and averaged AUC, but those terms are usually interchangeable. I was confused by your statement based on that.

  6. #6
    TS Contributor
    Points: 40,771, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Downloads
    gianmarco's Avatar
    Location
    Italy
    Posts
    1,370
    Thanks
    233
    Thanked 302 Times in 226 Posts

    Re: cross-validated AUC

    @hlsmith:
    when I used 'accuracy' I was referring at the output of the DAAG package (command: CVbinary): it returns the accuracy which is the percentage of the correctly classified cases out of the total of the cases. This can be easily calculated from a confusion matrix. In this case, the accuracy depends on the cutoff threshold on probability. As far as I understand, AUC does not depends on a specific cutoff value and, indeed, in the dataset I was playing with, accuracy (50% cutoff point) was 85% while AUC was 0.917.
    http://cainarchaeology.weebly.com/

  7. #7
    Human
    Points: 12,686, Level: 73
    Level completed: 59%, Points required for next Level: 164
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,363
    Thanks
    455
    Thanked 462 Times in 402 Posts

    Re: cross-validated AUC

    Are you talking about AUC = Area Under Curve? in the context of ROC-curves?

  8. #8
    TS Contributor
    Points: 40,771, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Downloads
    gianmarco's Avatar
    Location
    Italy
    Posts
    1,370
    Thanks
    233
    Thanked 302 Times in 226 Posts

    Re: cross-validated AUC

    @Greta: Yes, I am.
    http://cainarchaeology.weebly.com/

  9. #9
    Points: 1,143, Level: 18
    Level completed: 44%, Points required for next Level: 57

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: cross-validated AUC

    Ciao, non so se posso aiutarti, ma l'argomento mi interessa e sto cercando di capirci qualcosa di più anche io.

    Se ho capito bene tu hai i tuoi 10 gruppi in cui hai spezzato il dataset e vorresti avere l'auc medio dei gruppi, che poi penso sia l'auc del modello crossvalidato, è giusto?

  10. #10
    Human
    Points: 12,686, Level: 73
    Level completed: 59%, Points required for next Level: 164
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,363
    Thanks
    455
    Thanked 462 Times in 402 Posts

    Re: cross-validated AUC

    What I am after is getting the distribution of AUC values
    How about the bootstrap?

  11. #11
    TS Contributor
    Points: 40,771, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Downloads
    gianmarco's Avatar
    Location
    Italy
    Posts
    1,370
    Thanks
    233
    Thanked 302 Times in 226 Posts

    Re: cross-validated AUC


    Yes Greta, that would be a possibility. The package cvAUC provides the option to calculate the auc in the context of k-fold CV (i.e., getting 10 auc and their average), but I di not manage to put it to work.
    http://cainarchaeology.weebly.com/

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats