+ Reply to Thread
Results 1 to 11 of 11

Thread: Test validity

  1. #1
    Points: 121, Level: 2
    Level completed: 43%, Points required for next Level: 29

    Posts
    9
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Test validity




    Hello friends,

    I'm really new at this, I'm in high school, and wanted your help to know how does someone check or calculate if a test is actually valid?

    For example, my teacher wanted to rule out all the people who will fail her test. She assumed that 10 students would pass and in reality 10 did. She also said 2 would pass but these 2 failed. She also predicted that 24 would fail but in fact these 24 passed. She also predicted 12 would fail and these 12 actually did fail.

    What do I need to calculate or check to evaluate if this screening tool is valid?

    Thank you for any advice you give me.

    Pammy

  2. #2
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Test validity

    You seemingly have all necessary information to calculate
    -> sensitivity, -> specifity, -> positive predictive power,
    -> negative predictive power of the test (with the given
    cutoff).

    With kind regards

    K.

  3. The Following User Says Thank You to Karabiner For This Useful Post:

    Pammy (02-03-2015)

  4. #3
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Test validity

    You may want to also calculate the accuracy using the referenced approach, via summing the A and D cell in your 2 x 2 classification table and dividing by the total number of tests. Also, if possible you could work towards placing 95% confidence intervals on this accuracy value.
    Stop cowardice, ban guns!

  5. The Following 2 Users Say Thank You to hlsmith For This Useful Post:

    Karabiner (02-03-2015), Pammy (02-03-2015)

  6. #4
    Points: 121, Level: 2
    Level completed: 43%, Points required for next Level: 29

    Posts
    9
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: Test validity

    Thank you for your response!
    So here are my calculations now.
    TP 846 FP 15
    FN 131 TP 9
    Sen 87% Spec 38% Pos PV 98% Neg PV 6% Prevalence 98% Pos Likelihood ratio 1.39 Neg Likelihood Ratio .36
    So how do I interpret these results to know if it's a good valid test? Still confused
    Thank you,
    Pammy


    Quote Originally Posted by Karabiner View Post
    You seemingly have all necessary information to calculate
    -> sensitivity, -> specifity, -> positive predictive power,
    -> negative predictive power of the test (with the given
    cutoff).

    With kind regards

    K.

  7. #5
    Points: 121, Level: 2
    Level completed: 43%, Points required for next Level: 29

    Posts
    9
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: Test validity

    Sorry I made a mistake with my calculations. Here are the corrected ones?

    So here are my calculations now.
    TP 846 FP 14
    FN 129 TP 11
    Sen 87% Spec 44% Pos PV 98% Neg PV 8% Prevalence 98% Pos Likelihood ratio 1.55 Neg Likelihood Ratio .30

    So how do I interpret these results to know if it's a good valid test?


    Quote Originally Posted by Pammy View Post
    Thank you for your response!
    So here are my calculations now.
    TP 846 FP 15
    FN 131 TP 9
    Sen 87% Spec 38% Pos PV 98% Neg PV 6% Prevalence 98% Pos Likelihood ratio 1.39 Neg Likelihood Ratio .36
    So how do I interpret these results to know if it's a good valid test? Still confused
    Thank you,
    Pammy

  8. #6
    Points: 121, Level: 2
    Level completed: 43%, Points required for next Level: 29

    Posts
    9
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: Test validity

    Thank you.
    I did per your guidance by summing A and D cell and dividing by total number and I got 86%
    Here's the rest
    Sensitivity = 87 % 95% CI: 84.48 % to 88.83 %
    Specificity = 44 % 95% CI: 24.43 % to 65.06 %
    Positive Likelihood Ratio = 1.55% 95% CI: 1.09 to 2.20
    Negative Likelihood Ratio = 0.30 95% CI: 0.19 to 0.48
    prevalence = 98 % 95% CI: 96.33 % to 98.38 %
    Positive Predictive Value = 98.37 % 95% CI: 97.28 % to 99.11 %
    Negative Predictive Value = 8 % 95% CI: 3.99 % to 13.63 %

    How do I interpret these numbers so I know I don't waste my time if this will be a useless test. How do I know if it's a good valid one to more forward with?

    Thank you!!!!!!
    Pammy

    Quote Originally Posted by hlsmith View Post
    You may want to also calculate the accuracy using the referenced approach, via summing the A and D cell in your 2 x 2 classification table and dividing by the total number of tests. Also, if possible you could work towards placing 95% confidence intervals on this accuracy value.

  9. #7
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Test validity

    This is difficult to say. It depends on what it's used for. There is
    not "the" validity - it depends on base rate, cutoff, and what
    one considers important (few false-negative versus few false-
    positive decisions). In your sample there are very few true
    negative cases (only 2,4%).

    Specifity is low, but screening tests need not be very specific.
    Don't know whether 0.87 is a satisfactory value for sensitivity
    in screenings, maybe you can do some literature search.

    Funny thing is, if you compare it to a strategy which says "all are
    positive", those strategy would result in 97,6% correct assessments,
    100% sensivity, and an increase of FP from n=15 to only n=24.
    I guess it's extremely difficult for any test to identify the 2,4%
    negatives correctely without producing high false-negative rates.

    Anyway, I suppose that just 0.09 for NPP is not in order for a
    diagnostic screening test which is supposed to accept an increased
    false-positive rate but not high false-negative rates.

    With kind regards

    K.

  10. #8
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Test validity

    Karabiner, can you add a little more on the following. I have seen this mentioned in decision curve analysis, but was curious on your calculations.


    Quote Originally Posted by Karabiner View Post
    Funny thing is, if you compare it to a strategy which says "all are
    positive", those strategy would result in 97,6% correct assessments,
    100% sensivity, and an increase of FP from n=15 to only n=24.
    I guess it's extremely difficult for any test to identify the 2,4%
    negatives correctely without producing high false-negative rates.
    Stop cowardice, ban guns!

  11. #9
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Test validity

    Karabiner, can you add a little more on the following. I have seen this mentioned in decision curve analysis, but was curious on your calculations.
    If I read the OP's table correctly, then among 1000 observations
    there were 14 false-positives (=true negatives) and 11 correct negatives
    (=true negatives). Therefore, only 25 out of 1000 cases were true negatives,
    975 were true positives. Predicting "positive" for all 1000 cases would be
    correct in 975 cases and wrong in 25 cases (accuracy 97,5%). The current
    129 false-negatives would no longer exist (i.e. sensitivity 100%), and the number
    of false-positives would only be that of the whole "false" group, which is 25.

    With kind regards

    K.

  12. The Following User Says Thank You to Karabiner For This Useful Post:

    hlsmith (02-03-2015)

  13. #10
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Test validity

    I did not do any math and assumed that is what you were referencing, but did not remote.y think their was actually 1,000 observations. So I got a little confused.
    Stop cowardice, ban guns!

  14. #11
    Points: 121, Level: 2
    Level completed: 43%, Points required for next Level: 29

    Posts
    9
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: Test validity


    Thank you all for your help. IT helped me out! BTW, you are correct total N = 1000.

    Have a great day!

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats