Study design? Diagnostic Errors

Hello all,

I am designing a study and wondering if you might help me with the design from a statistical perspective please?

I am studying overnight errors by radiology residents. For instance, an xray may be interpretated as normal at 3am, then in the morning the attending reads the study and finds a fracture.

I have designed an intervention which is a practice test that residents can take to train themselves. The test consists of a number of questions (around 50) which present the user with an image and multiple choice. (I am using the cases from overnight missed findings over the past year.)

I am looking to show that by taking the test, the resident improves their performance. My question is: how can I show this in a statistically appropriate way?

My thoughts of possible ways:
1) Randomize the order of the questions, show that the correct answer is more likely to be chosen if it is taken later in the exam
2) Show that the first half of the questions have a lower percent correct than the last half
3) ???

Any idea how I should go about this? Or if this incorrect forum to ask, my apologies, where else should I seek help?

Thank you!



Fortran must die
Why don't you randomly assign two groups of individuals one who take the test and one who do not. Than have them do the evaluations. Then use ANOVA (or regression) to see if their scores vary signficantly.

Your comments suggest that time, or some other factor, causes people to change their evaluation of records (leading to errors). The method I suggested above would deal with whether people were making errors at a given moment. You would have to seperately show this influenced reduction of errors between time periods.