# Weighted Cohen Kappa (R)

#### hlsmith

##### Not a robit
Well I am attempting to get some code ready to calculated two rater Cohen Kappa inter-rater reliability with Fleiss-Cohen weights with 99% CIs. I am trying to use R and figured I would post this to see if you all can save me some time. Toy dataset:

Code:
NB <- read.table(header = TRUE, text = "
Rater1 Rater2
3      3
3      6
3      4
4      6
5      2
5      4
2      2
3      4
5      3
2      3
2      2
6      3
1      3
5      3
2      2
2      2
1      1
2      3
4      3
3      4
")
It is having trouble with the weights part, so I am trying to define the weights in a confusion matrix and feed them into the function. The following is the code that is bring up an issue.
Code:
install.packages("DescTools")
library(DescTools)
CohenKappa (NB$Rater1, NB$Rater2,
weights = c("Fleiss-Cohen"),
conf.level = c(0.01))
See example here: https://rdrr.io/cran/DescTools/man/CohenKappa.html

I will update my progress as I move forward. But also if you know of another package that can easily get the Kappa coefficient and confidence intervals please recommend and provide code if possible.

Thanks.

#### Dason

What is your issue? If it's an error please provide the output of the error message. If it's just output that doesn't make sense post that and tell us why it doesn't make sense. Thanks.

#### hlsmith

##### Not a robit
Error, which should be reproducible per above code. See following:

Error in CohenKappa(NB$Rater1, NB$Rater2, weights = c("Fleiss-Cohen"), :
Vector interface for weighted Kappa is not supported. Provide confusion matrix.

So I am trying other packages as well as trying to create a confusion matrix using the Fleiss-Cohen weightt formula, which in my mind is pseudo autoregressive.

$$1 - \frac{(i - j)^2}{(r - 1)^2}$$

Sorry I couldn't quit figure out the math tag above.

#### spunky

##### Doesn't actually exist
Found this in the documentation:

"(Note, that the vector interface can not be used together with weights)"

So I guess you're stuck using the matrix of weights as opposed to the default calculations it does? I looked at the documentation and the original article by Fleiss, Cohen & Everitt:

Fleiss, J.L., Cohen, J., and Everitt, B.S. (1969), Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 332-327.

And it seems doable to do by hand? Or maybe you can take a dive in their github to get the necessary elements to do the calculations yourself. I find myself doing that more often than not these days.

#### hlsmith

##### Not a robit
@spunky you are not suppose to reply back using word, but explicit code solution.

#### spunky

##### Doesn't actually exist
@spunky you are not suppose to reply back using word, but explicit code solution.
Dason once offered this as an all-purpose, code-based solution to inference and stuff

Code:
runif(1,0,1)
Perhaps this may be of use to you now?

#### hlsmith

##### Not a robit
@spunky

Code:
Spunkys_Response = c("Not Good Enough",
"Is Horrible",
"Perfect",
"Bryan was Right",
"Insufficient",
"Lacks Love",
"I Wish Hamid was Here")
sample(Spunkys_Response, 1)

Last edited:

#### hlsmith

##### Not a robit
Well for posterity: I used the below function with a bootstrap. Also below that is the SAS code, which won't kick out estimates if the table form of data are not symmetrical.

Code:
NB <- read.table(header = TRUE, text = "
Rater1 Rater2
3      3
3      6
3      4
4      6
5      2
5      4
2      2
3      4
5      3
2      3
2      2
6      3
1      3
5      3
2      2
2      2
1      1
2      3
4      3
3      4
5      5
")
NB
ck.boot <-function(NB,s=1:nrow(NB)) {kappa2(NB[s,],'squared')$value} library(boot) ckb <- boot(NB,ck.boot,R=1000) ckb hist(ckb$t)
boot.ci(ckb, type='bca')

Code:
data kap;
input R1 R2;
datalines;
3      3
3      6
3      4
4      6
5      2
5      4
2      2
3      4
5      3
2      3
2      2
6      3
1      3
5      3
2      2
2      2
1      1
2      3
4      3
3      4
5      5
;

proc freq data=kap;
table r1*r2 / agree (wt=fc);
run;