Test validity

Pammy

New Member
#1
Hello friends,

I'm really new at this, I'm in high school, and wanted your help to know how does someone check or calculate if a test is actually valid?

For example, my teacher wanted to rule out all the people who will fail her test. She assumed that 10 students would pass and in reality 10 did. She also said 2 would pass but these 2 failed. She also predicted that 24 would fail but in fact these 24 passed. She also predicted 12 would fail and these 12 actually did fail.

What do I need to calculate or check to evaluate if this screening tool is valid?

Thank you for any advice you give me.

Pammy
 

Karabiner

TS Contributor
#2
You seemingly have all necessary information to calculate
-> sensitivity, -> specifity, -> positive predictive power,
-> negative predictive power of the test (with the given
cutoff).

With kind regards

K.
 

hlsmith

Omega Contributor
#3
You may want to also calculate the accuracy using the referenced approach, via summing the A and D cell in your 2 x 2 classification table and dividing by the total number of tests. Also, if possible you could work towards placing 95% confidence intervals on this accuracy value.
 

Pammy

New Member
#4
Thank you for your response!
So here are my calculations now.
TP 846 FP 15
FN 131 TP 9
Sen 87% Spec 38% Pos PV 98% Neg PV 6% Prevalence 98% Pos Likelihood ratio 1.39 Neg Likelihood Ratio .36
So how do I interpret these results to know if it's a good valid test? Still confused :(
Thank you,
Pammy


You seemingly have all necessary information to calculate
-> sensitivity, -> specifity, -> positive predictive power,
-> negative predictive power of the test (with the given
cutoff).

With kind regards

K.
 

Pammy

New Member
#5
Sorry I made a mistake with my calculations. Here are the corrected ones?

So here are my calculations now.
TP 846 FP 14
FN 129 TP 11
Sen 87% Spec 44% Pos PV 98% Neg PV 8% Prevalence 98% Pos Likelihood ratio 1.55 Neg Likelihood Ratio .30

So how do I interpret these results to know if it's a good valid test?


Thank you for your response!
So here are my calculations now.
TP 846 FP 15
FN 131 TP 9
Sen 87% Spec 38% Pos PV 98% Neg PV 6% Prevalence 98% Pos Likelihood ratio 1.39 Neg Likelihood Ratio .36
So how do I interpret these results to know if it's a good valid test? Still confused :(
Thank you,
Pammy
 

Pammy

New Member
#6
Thank you.
I did per your guidance by summing A and D cell and dividing by total number and I got 86%
Here's the rest
Sensitivity = 87 % 95% CI: 84.48 % to 88.83 %
Specificity = 44 % 95% CI: 24.43 % to 65.06 %
Positive Likelihood Ratio = 1.55% 95% CI: 1.09 to 2.20
Negative Likelihood Ratio = 0.30 95% CI: 0.19 to 0.48
prevalence = 98 % 95% CI: 96.33 % to 98.38 %
Positive Predictive Value = 98.37 % 95% CI: 97.28 % to 99.11 %
Negative Predictive Value = 8 % 95% CI: 3.99 % to 13.63 %

How do I interpret these numbers so I know I don't waste my time if this will be a useless test. How do I know if it's a good valid one to more forward with?

Thank you!!!!!!
Pammy

You may want to also calculate the accuracy using the referenced approach, via summing the A and D cell in your 2 x 2 classification table and dividing by the total number of tests. Also, if possible you could work towards placing 95% confidence intervals on this accuracy value.
 

Karabiner

TS Contributor
#7
This is difficult to say. It depends on what it's used for. There is
not "the" validity - it depends on base rate, cutoff, and what
one considers important (few false-negative versus few false-
positive decisions). In your sample there are very few true
negative cases (only 2,4%).

Specifity is low, but screening tests need not be very specific.
Don't know whether 0.87 is a satisfactory value for sensitivity
in screenings, maybe you can do some literature search.

Funny thing is, if you compare it to a strategy which says "all are
positive", those strategy would result in 97,6% correct assessments,
100% sensivity, and an increase of FP from n=15 to only n=24.
I guess it's extremely difficult for any test to identify the 2,4%
negatives correctely without producing high false-negative rates.

Anyway, I suppose that just 0.09 for NPP is not in order for a
diagnostic screening test which is supposed to accept an increased
false-positive rate but not high false-negative rates.

With kind regards

K.
 

hlsmith

Omega Contributor
#8
Karabiner, can you add a little more on the following. I have seen this mentioned in decision curve analysis, but was curious on your calculations.


Funny thing is, if you compare it to a strategy which says "all are
positive", those strategy would result in 97,6% correct assessments,
100% sensivity, and an increase of FP from n=15 to only n=24.
I guess it's extremely difficult for any test to identify the 2,4%
negatives correctely without producing high false-negative rates.
 

Karabiner

TS Contributor
#9
Karabiner, can you add a little more on the following. I have seen this mentioned in decision curve analysis, but was curious on your calculations.
If I read the OP's table correctly, then among 1000 observations
there were 14 false-positives (=true negatives) and 11 correct negatives
(=true negatives). Therefore, only 25 out of 1000 cases were true negatives,
975 were true positives. Predicting "positive" for all 1000 cases would be
correct in 975 cases and wrong in 25 cases (accuracy 97,5%). The current
129 false-negatives would no longer exist (i.e. sensitivity 100%), and the number
of false-positives would only be that of the whole "false" group, which is 25.

With kind regards

K.
 

hlsmith

Omega Contributor
#10
I did not do any math and assumed that is what you were referencing, but did not remote.y think their was actually 1,000 observations. So I got a little confused.