Bootstrapping 0.632+ AUC with fixed test sets

Mkkl

New Member
#1
Hi,

I want to compare different binary classifiers in terms of their AUC values on test sets.
I have data over one year with about 2000 observations each month. Now I would like to use the data in each of the last three months as separate test sets while using the months prior to a particular test month as my training data. The reasons for this is that this will also happen in practice.

Instead of simply comparing the AUC for each model I wish to compare them while introducing some variance in the training set. Since, my test sets are fixed, I thought of using bootstrapping to take B bootstrap samples of the training sets and then evaluate the classifiers on all these B training sets. When looking into the 0.632+ bootstrap estimator by Efron I concluded that they use the observations in the original training set, which have not been included in a particular bootstrap sample, as the test set. However, as my test sets are fixed I would like to avoid this.

Does someone have an idea how to bootstrap while keeping your test sets fixed?