Would this be a good exam question?

FelixBossuyt

New Member
For statistical modelling and datamining, we have an assignment where we have to create a good exam question. I managed to come up with this. Would you consider it a good exam question or do you think it misses the point?
++ 1
What measure for classification and diagnostic testing should be increased in order to minimize a Type II error?
+ 1
+ 0
Precision
+ 0
Null rate
+ 0
Specificity

hlsmith

Less is more. Stay pure. Stay poor.
I feel like the response options are a hodgepodge.

So type II error - accepting the null when it is false. Easy answer is 1-beta or power, OK. If you have precision that would work too right. Also, in diagnostics I couldn't be convinced that someone would not be able to justify increasing Specificity would assist with accepting the null. And I have no idea what a null rate is.

So this isn't a good questions at all in my opinion. I get that most people would select "power", but can you clearly rule out the other options, no!

FelixBossuyt

New Member
I feel like the response options are a hodgepodge.

So type II error - accepting the null when it is false. Easy answer is 1-beta or power, OK. If you have precision that would work too right. Also, in diagnostics I couldn't be convinced that someone would not be able to justify increasing Specificity would assist with accepting the null. And I have no idea what a null rate is.

So this isn't a good questions at all in my opinion. I get that most people would select "power", but can you clearly rule out the other options, no!

FelixBossuyt

New Member
@FelixBossuyt - happy to help. What did you mean by null rate?
That was the error rate that a classifier that always classiefed no would have. I have taken your feedback into consideration and changed it to this:
++ 1
What measure for classification and diagnostic testing should be increased in order to minimize the probabilty of incorrectly retaining the null hypothesis, also known as the Type II error?
+ 1
Sensitivity
+ 0
Precision
+ 0
Null classifier
+ 0
Specificity

Explanation:
I chose this question because it assesses the student's knowledge about the terminology used to describe the confusion matrix. The question is formulated in such a way that the student needs to understand the underlying theory of the terms rather than learning the terms by heart. Sensitivity or true positive rate measures the proportion of actual positives that are correctly identified. The Type II error or false negative rate is the error that is made when an truly positive % outcome has been incorrectly predicted as negative by our model. By consequence if the sensitivity increases and more true positive predictions are made, the false negative predictions decrease. In contrast Specificity is the percentage of false outcomes that are correctly predicted. Precision is ratio of correctly predicted positive outcomes to all the predicted positive outcome. A null classifier is a predictor that always predicts a negative outcome.