Weighted Cohen Kappa (R)

hlsmith

Not a robit
#1
Well I am attempting to get some code ready to calculated two rater Cohen Kappa inter-rater reliability with Fleiss-Cohen weights with 99% CIs. I am trying to use R and figured I would post this to see if you all can save me some time. Toy dataset:

Code:
NB <- read.table(header = TRUE, text = "
              Rater1 Rater2
              3      3
              3      6
              3      4
              4      6
              5      2
              5      4
              2      2
              3      4
              5      3
              2      3
              2      2
              6      3
              1      3
              5      3
              2      2
              2      2
              1      1
              2      3
              4      3
              3      4
")
It is having trouble with the weights part, so I am trying to define the weights in a confusion matrix and feed them into the function. The following is the code that is bring up an issue.
Code:
install.packages("DescTools")
library(DescTools)
CohenKappa (NB$Rater1, NB$Rater2,
               weights = c("Fleiss-Cohen"),
              conf.level = c(0.01))
See example here: https://rdrr.io/cran/DescTools/man/CohenKappa.html

I will update my progress as I move forward. But also if you know of another package that can easily get the Kappa coefficient and confidence intervals please recommend and provide code if possible.

Thanks.
 

Dason

Ambassador to the humans
#2
What is your issue? If it's an error please provide the output of the error message. If it's just output that doesn't make sense post that and tell us why it doesn't make sense. Thanks.
 

hlsmith

Not a robit
#3
Error, which should be reproducible per above code. See following:

Error in CohenKappa(NB$Rater1, NB$Rater2, weights = c("Fleiss-Cohen"), :
Vector interface for weighted Kappa is not supported. Provide confusion matrix.


So I am trying other packages as well as trying to create a confusion matrix using the Fleiss-Cohen weightt formula, which in my mind is pseudo autoregressive.


\(1 - \frac{(i - j)^2}{(r - 1)^2}\)

Sorry I couldn't quit figure out the math tag above.
 

spunky

Doesn't actually exist
#4
Found this in the documentation:

"(Note, that the vector interface can not be used together with weights)"

So I guess you're stuck using the matrix of weights as opposed to the default calculations it does? I looked at the documentation and the original article by Fleiss, Cohen & Everitt:

Fleiss, J.L., Cohen, J., and Everitt, B.S. (1969), Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 332-327.

And it seems doable to do by hand? Or maybe you can take a dive in their github to get the necessary elements to do the calculations yourself. I find myself doing that more often than not these days.
 

hlsmith

Not a robit
#7
@spunky

Code:
Spunkys_Response = c("Not Good Enough",
                    "Makes Me Made",
                    "Is Horrible",
                    "Perfect",
                    "Bryan was Right",
                    "Insufficient",
                    "Lacks Love",
                    "I Wish Hamid was Here")
sample(Spunkys_Response, 1)
 
Last edited:

hlsmith

Not a robit
#8
Well for posterity: I used the below function with a bootstrap. Also below that is the SAS code, which won't kick out estimates if the table form of data are not symmetrical.

Code:
NB <- read.table(header = TRUE, text = "
              Rater1 Rater2
                 3      3
                 3      6
                 3      4
                 4      6
                 5      2
                 5      4
                 2      2
                 3      4
                 5      3
                 2      3
                 2      2
                 6      3
                 1      3
                 5      3
                 2      2
                 2      2
                 1      1
                 2      3
                 4      3
                 3      4
                 5      5
                 ")
NB
ck.boot <-function(NB,s=1:nrow(NB)) {kappa2(NB[s,],'squared')$value}
library(boot)
ckb <- boot(NB,ck.boot,R=1000)
ckb
hist(ckb$t)
boot.ci(ckb, type='bca')

Code:
data kap;
input R1 R2;
datalines;
3      3
3      6
3      4
4      6
5      2
5      4
2      2
3      4
5      3
2      3
2      2
6      3
1      3
5      3
2      2
2      2
1      1
2      3
4      3
3      4
5      5
;

proc freq data=kap;
table r1*r2 / agree (wt=fc);
run;