Multi-rater agreement statistic

Hi all,

This is my first post question.

I have a question on multirater agreement. Raters are given a description of the condition of the each of the same 10 patients and are then asked at what time of the day (Overnight morning afternoon evening none) would their condition be of most concern. They are also asked at what time of the day is their condition be of the second most concern. Thus every rater rated each of the 10 patients into the time of most concern and the time of the second most concern.

Is their a way to calculate a measure of agreement between the raters but not individually for the most concern and the second most concern but by giving a weight to the second most concern (say .5)?


Hi Masteras,

Thanks. Yes there are 5 possible answers.

I now have a solution; making 20 unique combinations of the 5 possible answers (overnight-morning, overnight-afternoon, morning-overnight.......) for the 1st and 2nd most concern time and finding the rater agreement for these. I will also, as sensitivity analysis, not use order (i.e. assuming the 1st and 2nd most concern are of equal value and thus have overnight-morning = morning -overnight, etc) and find the inter-rater agreement for these.

Thanks everyone.