Trying to compare a study group's actual outcomes to predicted risk using individual risk score estimates

jamss

New Member
#1
Hello. I'm trying to understand if it is possible/statistically sound to compare the percentage of actual complications to some aggregate of the group's individual risk calculated from a risk calculator.

For example, I have a 3 dogs undergoing treatment for clots. Their individual risk estimates of bleeding using a risk calculator were, 40%, 50%, and 60%. Three of them bled. How do I compare my outcome of 100% bleeding to the risk calculators estimate?

I have tried to look online. The closest test I've seen is the Brier test, but I would like to do the opposite of the brier test. I want to compare my performance to the risk calculated for each puppy that I've treated.

Thank you for your time. I appreciate any help.
 
Last edited:
#2
I'm not sure if this is what you want, but if the calculator is correct, then the probability of getting all three bleeding is 0.4x0.5x0.6 =0.12. So in a large number of similar situations, if the calculator is accurate then about 12% of the time there will be three bleeders out of three. This isn't strong evidence that the calculator is wrong. You might have just been unlucky.
 
#3
Hello! Thank you for the response. My question is whether I can use the risk calculator estimate as my population mean. I want to pool the values from the risk calculator as the standard to which I'm comparing my observed outcomes. However, I'm concerned that that may be double dipping.
 

hlsmith

Not a robit
#4
How many actual obs do you have? Have you looked at the calibration curve plot?

Side note, if your sample different from the sample the tool was built on? If your sample is not a random sample of the population the sample the tool was built on it may have another feature present that wasnt found and used in the model building process.