Statistical significance

C

Carcassonne

New Member
#1
#1
Hi,

I have the following problem and I would need some piece of advice to solve it:
I have a group of 28 patients (17 survived, 11 died) and I am trying to find out if some of quantities are capable of discriminating survivors from non-survivors with a ROC analysis.
For all the considered quantities the AUC values are close to 0.5 with p-values >0.05. Can I conclude that none of the analysed quantities is a good discriminant for the outcome? In other words, the high p-value is enough to make this conclusion or could this result just be due to the low number of patients in my sample? In case the result is due to the small sample size, how can I calculate the size of the sample that I would need to draw this conclusion?
Thank you very much for your help!

Best regards,
 
hlsmith

hlsmith

Less is more. Stay pure. Stay poor.
#2
#2
Did you use multiple logistic regression to do this?


Most people will say that with only 11 events, your sample size may only support 1 predictor at most. How many predictors did you examine and how were they formatted (e.g., binary variables), and did you test them one at a time? A p-value > 0.5 for an AUC value just means the variable is not any better then chance, and since they were close to 0.5, that doesn't bode well for it just being a small sample size thing.


Since your outcome was death status, was it not feasible to examine data with survival model which would take into account time?


A post hoc sample size / power calculation could be frown upon, if it directs you to collect more data to just try and get significance.




If variables are truly independent and you are not looking to increase sample size or kept the study going, you could easily get an idea of power issues by just calculating it for bivariate fisher test power analyses (if IVs are categorical), since you have a finite sample size. So run a power analysis for Fisher exact test for a outcome and variable.


Another option would be to run simulations to determine what sample size would be need for distinguishing say an AUC value of 0.55 from 0.50.
 
C

Carcassonne

New Member
#3
#3
hlsmith said:
Did you use multiple logistic regression to do this?


Most people will say that with only 11 events, your sample size may only support 1 predictor at most. How many predictors did you examine and how were they formatted (e.g., binary variables), and did you test them one at a time? A p-value > 0.5 for an AUC value just means the variable is not any better then chance, and since they were close to 0.5, that doesn't bode well for it just being a small sample size thing.


Since your outcome was death status, was it not feasible to examine data with survival model which would take into account time?


A post hoc sample size / power calculation could be frown upon, if it directs you to collect more data to just try and get significance.




If variables are truly independent and you are not looking to increase sample size or kept the study going, you could easily get an idea of power issues by just calculating it for bivariate fisher test power analyses (if IVs are categorical), since you have a finite sample size. So run a power analysis for Fisher exact test for a outcome and variable.


Another option would be to run simulations to determine what sample size would be need for distinguishing say an AUC value of 0.55 from 0.50.
Click to expand...


Hi,

Thank you very much for your answer!

I am interested in assessing whether or not a certain variable is able to predict the outcome. Even a negative result (i.e. the variable cannot predict the outcome) is an interesting result for me. However, I would like to be sure that the negative result I obtain is not merely due to the small sample size.
I tested each variable separately and used a logistic regression (not a multiple one, since it would not make so much sense for my problem to combine the variables together in the same model).
The variables I tested were 8, all of them were continuous variables.

I have the day of death of these patients and I could perform a survival analysis, but I am interested in knowing whether or not the variable under consideration is able to predict the outcome and, in the positive case (statistical significance), I would like to know the threshold for this variable having the highest sensitivity and specificity (optimal criterion). This is the reason why I decided to perform an ROC analysis.
Do you think that a survival analysis would be a better option?

Unfortunately, the number of patients cannot change; I am not able to collect more data.
I am not familiar at all with sample size calculations and power issues, but I tried to run a test for the sample size and AUC values in the software I use (MedCalc). My results are in the attachment. I am not sure if I performed the test correctly. Results seem to indicate that to differentiate between an AUC value of 0.55 and 0.5 I would need more than 600 patients (and this is not doable for me). What is your opinion?
Thank you very much again!
 
CB

CB

Super Moderator
#4
#4
Looks like your messages were being flagged by our spam software (likely due to the use of attachments, given that you're a new-ish user). I've released your most recent post and deleted the two duplicates. Sorry about the delay.
 
hlsmith

hlsmith

Less is more. Stay pure. Stay poor.
#5
#5
What program did you use for the regression?

So did you dichotomize the continuous variable and that is where the 0.55 is coming from?

Just curious what your SEN, SPEC values are for the variable.

If you can post your 2x2 confusion matrix I may play around with your numbers. Surprisingly I don't run many survival curves, but given a linear relationship between the continuous variable and outcome, you could get probabilities for values of continuous var for the outcome and hazard ratios which are like relative risks. Depends on what your question is???
 
C

Carcassonne

New Member
#6
#6
Hi hlsmith,
I posted my reply here with all my calculations attached a few days ago but it was stopped again. I tried to contact both CowboyBear directly and the administrators multiple times in order to fix this problem but I haven't got any reply yet. I hope they will make my post visible soon.
Thank you for your help!
 
CB

CB

Super Moderator
#7
#7
There's no post showing here waiting to be released - whatever the problem is may be something other than the spam filter?
 
You must log in or register to reply here.