Interrater Reliability. Is MORE better?

[bitor]

Which design is more powerful: testing interrater reliability with 2 judges or 4 judges? (citations appreciated)

 

[Jake]

More data (judges) is always better.

 

[Dason]

Agreed. I don't see how there would be any contest between those two designs.

 

[hlsmith]

More judges would help to minimize sampling issues in acquiring a representative sample of judges, i.e., if you had two judges and one was biased in some way they may weight the conclusion the wrong direction of the real truth.