kappa

Hi, For my research, I need to calculate the inter rater reliability. There are 4 observers, who each observe the same objects and score it for a total of 7 variables. The variables are of ordinal level, but not each variable has the same amount of outcome variables. Which inter rater...

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability of the 2 modes of administration. In other words: does the electronic version of the...

I am writing a research paper. We asked the same patients to fill out a form at two different times. Data is non-normal. I used Spearman's correlation, which gave me a moderate correlation between the two scores. I also used weighted kappa and this too gave me a moderate agreement between both...

Hi! I have two raters (patient and physician) who rate the a condition at several visits. I would like to account for inter-individual correlation when I estimate kappa. And I want to assess of there is a trend in kappa. For continuous data, the the cccrm package (R) could work. I´m unsure...

Hi There, First time user here. I am a Master's student of Primatology, I wasn't sure Psychology Stats was the way to go but it seems to fit better than Biostatistics for my question. My thesis is the gestural communication and social cognition of captive northern white-cheeked gibbons...

I need to find the interrater reliability between 3 raters. A nurse (non-expert) performed 60 ultrasound scans and categorised patients into 3 ordinal and also categorical variables (hypovolemic, euvolemic, hypervolemic) according to the scans. The recorded scans were then subsequently...

Hello All, Thank you in advance if you are able to provide some information on my question. I am attempting to run Cohen's kappa for inter-rater agreement in SPSS. I have 2 raters; however, there are multiple variables (20+) that they are each rating. My question is how do I go about setting...

Hi, I am working with a stratified sample from a survey (a complex sample). I just tried to measure the agreement between two variables with the kappa test but it seems there is no way to take the nature of the sample into account in SPSS. We know the Kappa value will be the same, but...

Hi all, This is my first post question. I have a question on multirater agreement. Raters are given a description of the condition of the each of the same 10 patients and are then asked at what time of the day (Overnight morning afternoon evening none) would their condition be of most...

Hello, I have three coders who have rated 22 dichotomous (present/absent) variables for 15 open-ended responses. Is it appropriate to use Fleiss Kappa here? And if so, is there a unique syntax for such a case? Thanks to whomever can help, PG BAH, Psychology

Hi all, I am trying to compare 2 instruments, A and B, against the gold standard. The measurement outcome is dichotomous. 2 different raters, R1 and R2, each uses instrument A and B to rate each subject. So the data looks something like: ID, A_R1, A_R2, B_R1, B_R2...

Disclaimer: Not psychological research, I know, but the type of statistics are more often used in psychology than in medicine, so I'm asking for help here. Background: I had 4 radiologists read CT scans of ~50 patients to assess whether a series of 'signs' were present or not. After a washout...

Hi everyone, I'm having some difficulty deciding how best to analyze the data from a project. I had 4 radiologist read CT scans looking for specific signs (mostly giving True/False answers) on two occasions separated by a washout period. The last two variables included a 'Questionable'...

I am aiming to check the inter-rater reliability of a scale using Cohen's Kappa. I have input the 5 scores as their own variables for Rater A and the same again for Rater B. (1A, 2A, 3A, 4A, 5A and 1B, 2B, 3B etc). In each variable, the scores range from 0-40 and are not categories - just...

I am trying to calculate inter-rater reliability scores for 10 survey questions-- most of which are binary (yes/no). The agreemeant level bettwen the two raters is 70-90% on nearly every question, however the Kappa score is often very poor (0.2- 0.4). Can this be right? Secondly, can you...

I have two observers who watched a series of videos looking for gestures. They recorded the onset-time of any gesture they observed. An agreement was scored when the two observers recorded a gesture onset within 1-second of the other observer. A disagreement was scored when the one observer...

Hi everyone: I really appreciate all of the great posts on here! I'm trying to wrap my head around a couple of things. First of all, a bit of background: I am trying to "validate" a questionnaire that was developed against a "gold standard": medical records. The variables are all...