+ Reply to Thread
Results 1 to 14 of 14

Thread: Inter-observer variability

  1. #1
    Points: 1,881, Level: 25
    Level completed: 81%, Points required for next Level: 19

    Posts
    30
    Thanks
    2
    Thanked 1 Time in 1 Post

    Inter-observer variability




    Dear all,

    I hope so much anyone here may help me with my question regarding inter-observer variability:

    3 observers measure the diameter of five different types of tumors
    I would like to calculate the inter-observer variability.

    which of my following approaches would be the best:

    1. comparing min, mean and max values of each observer with an ANOVA
    2. mean of the following: difference of observer1-observer2; difference of observer1-observer3; difference of observer2-observer3
    3. SD of mean inter-observer difference for each tumor

    any other suggestions?

    Best wishes, and thank you a lot for your help
    Last edited by med1234; 07-01-2011 at 01:30 PM.

  2. #2
    Points: 1,881, Level: 25
    Level completed: 81%, Points required for next Level: 19

    Posts
    30
    Thanks
    2
    Thanked 1 Time in 1 Post

    Re: Inter-observer variability

    hope so much anyone can help with my question... I have now edited the post and provided some more detailed possibilities to solve the problem, that seem to be reasonable for me. are they correct, or any other suggestions?

  3. #3
    TS Contributor
    Points: 22,399, Level: 93
    Level completed: 5%, Points required for next Level: 951
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Inter-observer variability

    i am still a little bit confused for what you're trying to get to...would measures of inter-rater reliablity be of any help? like: http://en.wikipedia.org/wiki/Inter-rater_reliability

  4. #4
    Points: 1,881, Level: 25
    Level completed: 81%, Points required for next Level: 19

    Posts
    30
    Thanks
    2
    Thanked 1 Time in 1 Post

    Re: Inter-observer variability

    I am so happy, about all the helping people in this forum. Thank you spanky, you have saved my weekend!
    yes --- inter-rater reliability seems to be what I am looking for.

    having 3 investigators, and 5 different tumors, which test would you use to speak about "how good" the reliability for their tumor measurements is. From the wikipedia article I do not really understand what might be the appropriate test for my situation (sorry I am not an statistician expert at all).

    As this kind of invesitgator's measurements are a bit difficult, I would like to find out, if the results of the three investigators are quite "similar", so that the method of measurement has a good reproducibility.

  5. #5
    TS Contributor
    Points: 22,399, Level: 93
    Level completed: 5%, Points required for next Level: 951
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Inter-observer variability

    ok... so if i understand you correctly then you have 15 different readings (each one of the 3 different researchers measures each one of the 5 different tumors)... and when they measure, what kind of scaling do they use? i mean, do they say something like: "this tumor is small, medium, or big.." or do they say somethign like: "this tumor is 0.something inches and that one is 0.somethign-else inches"?? it kinda helps out the way in which the data is collected before selecting one particular method over the other...

  6. #6
    Points: 1,881, Level: 25
    Level completed: 81%, Points required for next Level: 19

    Posts
    30
    Thanks
    2
    Thanked 1 Time in 1 Post

    Re: Inter-observer variability

    they measure the diameter in inches (therefore it's not a categorical value as small, medium or large, as far as I understand the terminology)

  7. #7
    TS Contributor
    Points: 22,399, Level: 93
    Level completed: 5%, Points required for next Level: 951
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Inter-observer variability

    and you got the terminology right. i think i'd go for the intra-class correlation coefficient in this case...

  8. #8
    Devorador de queso
    Points: 95,781, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,933
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Inter-observer variability

    Quote Originally Posted by med1234 View Post
    I am so happy, about all the helping people in this forum. Thank you spanky, you have saved my weekend!
    O_O

    One little letter makes a world of difference.

  9. #9
    TS Contributor
    Points: 22,399, Level: 93
    Level completed: 5%, Points required for next Level: 951
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Inter-observer variability

    Quote Originally Posted by Dason View Post
    One little letter makes a world of difference.
    i know... but it made me laugh so hard i kinda had to just let it there...

  10. #10
    Devorador de queso
    Points: 95,781, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,933
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Inter-observer variability

    Made me laugh quite a bit too. My wife looked at me funny.

  11. #11
    Points: 1,881, Level: 25
    Level completed: 81%, Points required for next Level: 19

    Posts
    30
    Thanks
    2
    Thanked 1 Time in 1 Post

    Re: Inter-observer variability

    spanky -> spunky... oh my god... I feel so sorry... :-) I apologize for the misspelling...

    Thank you a lot for help....

    As my stat. software (Graphpad Prism for Mac) does not support to calculate the intraclass correlation coefficient (as far as I have understood it is not the same as performing an ANOVA or a standard correlation analysis) I have looked for another possibility to calculate it. And I have found this excel sheet: newstats.org/xICC.xls

    For me it is not really clear how to fill out this sheet.
    N subjects = 5 ? (as there are 5 different tumors to measure)

    between SD ??? (SD between subjects??? however as there are three measurements for each tumor, is it the SD of the mean of all tumors)

    within SD ??? (SD within subjects, would it be the mean of all five inter-observer SD?)

    deg of freedom --> no idea what this is in my case with three observers and five tumors

    conf level --> what should I choose...

    Sorry a lot, in the end I will become the "spanky" in this forum, if I have such a lot of beginners questions...

    THANK YOU VERY VERY MUCH!!!!!

  12. #12
    TS Contributor
    Points: 22,399, Level: 93
    Level completed: 5%, Points required for next Level: 951
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Inter-observer variability

    uhmm... ok. there's a limit to how much i can help you here because my computer at home does not support excel and, hence, i can't really see what they're asking for... but from what you described there i think i know where they are going with this. you need to perform a repeated-measures ANOVA on your data and, from there, get the between-groups mean square, within-groups mean square and the degrees of freedom which you later put on that spreadsheet and get your ICC (IntraClassCorrelation). the problem is that it has been years since last time i did one "by hand" so to speak and i'm an R/Minitab user (and intro-SPSS but that's not going all that well, hehehehe...).

    i am going to direct you to an article: Shrout, Patrick E. and Fleiss, Joseph L. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 1979, 86, 420-3428. Shrout & Fleiss are the guys who popularised the ICC so to speak (because there are many of them) and you should be able to choose which kind of ICC you'll need that better helps you with your data (there's like 5 or 7 different ICCs out there and they all work in different scenarios...)

    anyways, i hope someone with access to excel can look into this because everytime someone on the internet asks for something, they could be asking for the raw sums-of-squares or the variance or the standardized variance (which could be the case in your spreadsheet, i dont know)...

    good luck!

  13. #13
    Points: 1,881, Level: 25
    Level completed: 81%, Points required for next Level: 19

    Posts
    30
    Thanks
    2
    Thanked 1 Time in 1 Post

    Re: Inter-observer variability

    thank you a lot... I have now read some wikipedia articles, your reference, and some other things I have found over google...

    I hope I have now found the correct solution to insert into the excel sheet (since Graphpad Prism does not offer to calculate ICC)

    N = 5 (5 different kind of tumors)
    between_SD = mean ((SD for measurements of observer1), (SD for measurements of observer2), (SD for measuremtns of observer3)
    within_SD = mean ((SD for 3 measurements of tumor1),(SD for 3 measurements of tumor2),....(SD for 3 measurements of tumor5)
    DEG of freedom = 2 (as we have 3 raters/judges)
    Conf level = 90% (seems reasonable?!)

    ICC = (between_SD^2-within_SD^2)/between_SD^2

    dear spunky, can you tell me of my solution is correct? I am not sure if I have done it right...

  14. #14
    TS Contributor
    Points: 22,399, Level: 93
    Level completed: 5%, Points required for next Level: 951
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Inter-observer variability


    Quote Originally Posted by med1234 View Post
    dear spunky, can you tell me of my solution is correct? I am not sure if I have done it right...
    neither do i... i would be tempted to say that you did not do it right because what you're after are the variance components attributable to the between-groups factor and the error (or within-groups) one... however, since i dont know what your excel sheet is doing, i dont know whether it can somehow use the standard deviation and some other info to obtain the variance components you're looking for.... or whether they use "standard deviation" to actually mean "mean square error" which is what i believe they are trying to convey... but, once again, i dont have excel so i have no idea what information they're asking you to provide.

    if i were you, i would download R (which is free) install the "psych" package and ask it to give you the intraclass correlation from your raw data... ANOVAs are laborious to work out by hand and they can get very tricky really fast...

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats