How do I calculate reliability for something that I do not have an "exact" answer for? For example, in an observational setting with 2 observers counting the number of times eye contact is given for 2 different participants (total of 4 observances). There is no absolute answer for eye contact. I was wondering what statistic is best to use, especially given that there are only 2 observers. I tried Cronbach's alpha and Pearson but these give me correlations in the negative range. I think this is only because there are 2 observers and not because the numbers are unreliable.