Reliability

[roy]

How do I calculate reliability for something that I do not have an "exact" answer for? For example, in an observational setting with 2 observers counting the number of times eye contact is given for 2 different participants (total of 4 observances). There is no absolute answer for eye contact. I was wondering what statistic is best to use, especially given that there are only 2 observers. I tried Cronbach's alpha and Pearson but these give me correlations in the negative range. I think this is only because there are 2 observers and not because the numbers are unreliable.

[noetsi]

Negative values can reflect reverse scored questions (although I can't see how this could occur in your example) or unusual variation. I have never heard it suggested that only two observers influence alpha. Any negative results make no substantive sense. I don't understand how you can have no "absolute answer" for eye contact, but I am sure the more vague answers could be the more difficult reliability would be. I am not sure how you even score results that have to be interpreted this way.

Why not create a result that does have a clear, absolute answer for the raters?