1. ## Inter-observer variability

Dear all,

I hope so much anyone here may help me with my question regarding inter-observer variability:

3 observers measure the diameter of five different types of tumors
I would like to calculate the inter-observer variability.

which of my following approaches would be the best:

1. comparing min, mean and max values of each observer with an ANOVA
2. mean of the following: difference of observer1-observer2; difference of observer1-observer3; difference of observer2-observer3
3. SD of mean inter-observer difference for each tumor

any other suggestions?

Best wishes, and thank you a lot for your help

2. ## Re: Inter-observer variability

hope so much anyone can help with my question... I have now edited the post and provided some more detailed possibilities to solve the problem, that seem to be reasonable for me. are they correct, or any other suggestions?

3. ## Re: Inter-observer variability

i am still a little bit confused for what you're trying to get to...would measures of inter-rater reliablity be of any help? like: http://en.wikipedia.org/wiki/Inter-rater_reliability

4. ## Re: Inter-observer variability

I am so happy, about all the helping people in this forum. Thank you spanky, you have saved my weekend!
yes --- inter-rater reliability seems to be what I am looking for.

having 3 investigators, and 5 different tumors, which test would you use to speak about "how good" the reliability for their tumor measurements is. From the wikipedia article I do not really understand what might be the appropriate test for my situation (sorry I am not an statistician expert at all).

As this kind of invesitgator's measurements are a bit difficult, I would like to find out, if the results of the three investigators are quite "similar", so that the method of measurement has a good reproducibility.

5. ## Re: Inter-observer variability

ok... so if i understand you correctly then you have 15 different readings (each one of the 3 different researchers measures each one of the 5 different tumors)... and when they measure, what kind of scaling do they use? i mean, do they say something like: "this tumor is small, medium, or big.." or do they say somethign like: "this tumor is 0.something inches and that one is 0.somethign-else inches"?? it kinda helps out the way in which the data is collected before selecting one particular method over the other...

6. ## Re: Inter-observer variability

they measure the diameter in inches (therefore it's not a categorical value as small, medium or large, as far as I understand the terminology)

7. ## Re: Inter-observer variability

and you got the terminology right. i think i'd go for the intra-class correlation coefficient in this case...

8. ## Re: Inter-observer variability

Originally Posted by med1234
I am so happy, about all the helping people in this forum. Thank you spanky, you have saved my weekend!
O_O

One little letter makes a world of difference.

9. ## Re: Inter-observer variability

Originally Posted by Dason
One little letter makes a world of difference.
i know... but it made me laugh so hard i kinda had to just let it there...

10. ## Re: Inter-observer variability

Made me laugh quite a bit too. My wife looked at me funny.

11. ## Re: Inter-observer variability

spanky -> spunky... oh my god... I feel so sorry... :-) I apologize for the misspelling...

Thank you a lot for help....

As my stat. software (Graphpad Prism for Mac) does not support to calculate the intraclass correlation coefficient (as far as I have understood it is not the same as performing an ANOVA or a standard correlation analysis) I have looked for another possibility to calculate it. And I have found this excel sheet: newstats.org/xICC.xls

For me it is not really clear how to fill out this sheet.
N subjects = 5 ? (as there are 5 different tumors to measure)

between SD ??? (SD between subjects??? however as there are three measurements for each tumor, is it the SD of the mean of all tumors)

within SD ??? (SD within subjects, would it be the mean of all five inter-observer SD?)

deg of freedom --> no idea what this is in my case with three observers and five tumors

conf level --> what should I choose...

Sorry a lot, in the end I will become the "spanky" in this forum, if I have such a lot of beginners questions...

THANK YOU VERY VERY MUCH!!!!!

12. ## Re: Inter-observer variability

uhmm... ok. there's a limit to how much i can help you here because my computer at home does not support excel and, hence, i can't really see what they're asking for... but from what you described there i think i know where they are going with this. you need to perform a repeated-measures ANOVA on your data and, from there, get the between-groups mean square, within-groups mean square and the degrees of freedom which you later put on that spreadsheet and get your ICC (IntraClassCorrelation). the problem is that it has been years since last time i did one "by hand" so to speak and i'm an R/Minitab user (and intro-SPSS but that's not going all that well, hehehehe...).

i am going to direct you to an article: Shrout, Patrick E. and Fleiss, Joseph L. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 1979, 86, 420-3428. Shrout & Fleiss are the guys who popularised the ICC so to speak (because there are many of them) and you should be able to choose which kind of ICC you'll need that better helps you with your data (there's like 5 or 7 different ICCs out there and they all work in different scenarios...)

anyways, i hope someone with access to excel can look into this because everytime someone on the internet asks for something, they could be asking for the raw sums-of-squares or the variance or the standardized variance (which could be the case in your spreadsheet, i dont know)...

good luck!

13. ## Re: Inter-observer variability

thank you a lot... I have now read some wikipedia articles, your reference, and some other things I have found over google...

I hope I have now found the correct solution to insert into the excel sheet (since Graphpad Prism does not offer to calculate ICC)

N = 5 (5 different kind of tumors)
between_SD = mean ((SD for measurements of observer1), (SD for measurements of observer2), (SD for measuremtns of observer3)
within_SD = mean ((SD for 3 measurements of tumor1),(SD for 3 measurements of tumor2),....(SD for 3 measurements of tumor5)
DEG of freedom = 2 (as we have 3 raters/judges)
Conf level = 90% (seems reasonable?!)

ICC = (between_SD^2-within_SD^2)/between_SD^2

dear spunky, can you tell me of my solution is correct? I am not sure if I have done it right...

14. ## Re: Inter-observer variability

Originally Posted by med1234
dear spunky, can you tell me of my solution is correct? I am not sure if I have done it right...
neither do i... i would be tempted to say that you did not do it right because what you're after are the variance components attributable to the between-groups factor and the error (or within-groups) one... however, since i dont know what your excel sheet is doing, i dont know whether it can somehow use the standard deviation and some other info to obtain the variance components you're looking for.... or whether they use "standard deviation" to actually mean "mean square error" which is what i believe they are trying to convey... but, once again, i dont have excel so i have no idea what information they're asking you to provide.

if i were you, i would download R (which is free) install the "psych" package and ask it to give you the intraclass correlation from your raw data... ANOVAs are laborious to work out by hand and they can get very tricky really fast...

 Tweet

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts