I need to find the interrater reliability between 3 raters.
A nurse (non-expert) performed 60 ultrasound scans and categorised patients into 3 ordinal and also categorical variables (hypovolemic, euvolemic, hypervolemic) according to the scans. The recorded scans were then subsequently assessed by 2 blinded experts (ultrasound specialists) and each categorised the scans into the 3 categories.
I assume I need to use either weighted kappa or Fleiss kappa for the analysis of IRR, but the fact that 2 out of 3 raters are experts makes the comparison difficult as they might agree more often, even if they could not see each others results, correct ?
Which method is the best choice ?
Fleiss kappa , weighted kappa, or ICC in SPSS ?
Thanks for your help !