Hello,
I have the question which measurement to use in order to calculate the inter-observer reliability in R once for an ordinal and once for a nominal data set.
Part1) Ordinal data set
2 observers gave scores on a scale from 1.0 to 5.0 with steps of 0.25
Example of data:
observ1 <-(c=(2.25, 3.0,2.75, 3.5,3.5))
observ2<-(c=(2.5,3.0,2.75,3.25,3.5))
a) Is Cohen´s Kappa a valid measurement for this type of data sets? If yes, how can I calculate this with R? I have tried cohen.kappa {psych} without success, and am quite confused about the existence of more kappas in the packages irr, vcd and even base. Which one do you recommend?
b) In case Cohen´s Kappa is not feasible: Which measurement do you recommend and again: where is it to be found in R?
c) To make it more tricky, I wanted to compare the measurement (e.g. Kappa) using the data set given, to the (improved) repeatability measure from a second run, which accepts a deviation of 0.25 between observers. How do I incorporate this second run in R?
Part 2) Nominal data set
2 observers, example data sets:
Observer1=(c=(11,8,12,10))
Observer2=(c=(10,8,12,10))
1) Which measurement for the inter-observer reliability do you recommend and how do I calculate it with R?
I thank you in advance for your recommendations and answers.
Please do not hesitate to ask for more details if necessary.
Manatis
I have the question which measurement to use in order to calculate the inter-observer reliability in R once for an ordinal and once for a nominal data set.
Part1) Ordinal data set
2 observers gave scores on a scale from 1.0 to 5.0 with steps of 0.25
Example of data:
observ1 <-(c=(2.25, 3.0,2.75, 3.5,3.5))
observ2<-(c=(2.5,3.0,2.75,3.25,3.5))
a) Is Cohen´s Kappa a valid measurement for this type of data sets? If yes, how can I calculate this with R? I have tried cohen.kappa {psych} without success, and am quite confused about the existence of more kappas in the packages irr, vcd and even base. Which one do you recommend?
b) In case Cohen´s Kappa is not feasible: Which measurement do you recommend and again: where is it to be found in R?
c) To make it more tricky, I wanted to compare the measurement (e.g. Kappa) using the data set given, to the (improved) repeatability measure from a second run, which accepts a deviation of 0.25 between observers. How do I incorporate this second run in R?
Part 2) Nominal data set
2 observers, example data sets:
Observer1=(c=(11,8,12,10))
Observer2=(c=(10,8,12,10))
1) Which measurement for the inter-observer reliability do you recommend and how do I calculate it with R?
I thank you in advance for your recommendations and answers.
Please do not hesitate to ask for more details if necessary.
Manatis