I am doing a study to investigate the interrater reliability of a rating scale.
I solicited 100 teachers to complete a rating scale on a child and then have another teacher complete the same rating scale on the same child. The rating scale uses ordinal ratings (i.e., 0, 1, 2, 3) and consists of 30 items. The rating scale is standardized and has standard (scaled scores) for two subtests and an overall index of the scores. I want to determine the interrater reliability of the scale’s subtests and index.
My questions are: (1) What intraclass correlation coefficient model is most appropriate for this study? (2) What type of agreement should I use?