Within study calculate statistical significance

#1
Hi

I have a binomial experiment whereby I’m comparing the diagnostic accuracy of Test 1 against Test 2.

Test 2 comprises of two parts (A and B), with both parts completed a binomial response is given. Yes there is the disease or No there isn’t a disease. However Test 2 is expensive to complete both parts (A+B). So Test 1 comprises of only completing part (A) of Test 2.

The aim of this research is to know is there a statistically significant difference in diagnostic accuracy of Test 1 in comparison to Test 2 when looking at disease type 1 and disease type 2.

I’ve started by comparing each participant’s results from Test 1 (part A) against Test 2 (A+B combined) in order to determine the True Positive, True Negative, False positive and False negative and gone on to work out the sensitivity, specificity and accuracy.

1570814969618.png

Which statistical test would be most appropriate to calculate if I want to know if there is a statistically significant difference in the diagnostic accuracy of the two test? I'm not sure if I can assume that any difference in accuracy would indicate a statistically significant difference?
 

Karabiner

TS Contributor
#2
The aim of this research is to know is there a statistically significant difference in diagnostic accuracy of Test 1 in comparison to Test 2 when looking at disease type 1 and disease type 2.
Are you sure? In such situations, I would not bother about statistical significance at all.
It is almost certain that the diagnostic accuracy of 2 tests is not absolutely the same.
Therefore, rejecting the null hypothesis ("difference of acurracy = 0.00000000000")
will just depend on a large enough sample size. On the other hand, non-signifiance
will just be due to insufficient sample size (low power). So what's the use of
"statistical significance" here?

With kind regards

Karabiner
 
#3
Hi thank you so much for your reply.
Not really, now you’ve explained it I agree I don’t think I should be testing for statistical significance. I was just a bit lost as to how to prove if it’s “OK” to just do test A rather than both parts of test B.

Would you mind explainIng how I would prove I have sufficient sample size?

Again thank you so much.
 

Karabiner

TS Contributor
#4
The appropriate analysis here would be McNemar's test, AFAICS; calculating a confidence interval for the difference in proportions
would then give you an impression of how precise your estimates are.

With kind regards

Karabiner