+ Reply to Thread
Results 1 to 9 of 9

Thread: Statistical significance

  1. #1
    Points: 3,490, Level: 36
    Level completed: 94%, Points required for next Level: 10

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Statistical significance




    Hi,

    I have the following problem and I would need some piece of advice to solve it:
    I have a group of 28 patients (17 survived, 11 died) and I am trying to find out if some of quantities are capable of discriminating survivors from non-survivors with a ROC analysis.
    For all the considered quantities the AUC values are close to 0.5 with p-values >0.05. Can I conclude that none of the analysed quantities is a good discriminant for the outcome? In other words, the high p-value is enough to make this conclusion or could this result just be due to the low number of patients in my sample? In case the result is due to the small sample size, how can I calculate the size of the sample that I would need to draw this conclusion?
    Thank you very much for your help!

    Best regards,

  2. #2
    Omega Contributor
    Points: 39,128, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,085
    Thanks
    402
    Thanked 1,194 Times in 1,155 Posts

    Re: Statistical significance

    Did you use multiple logistic regression to do this?


    Most people will say that with only 11 events, your sample size may only support 1 predictor at most. How many predictors did you examine and how were they formatted (e.g., binary variables), and did you test them one at a time? A p-value > 0.5 for an AUC value just means the variable is not any better then chance, and since they were close to 0.5, that doesn't bode well for it just being a small sample size thing.


    Since your outcome was death status, was it not feasible to examine data with survival model which would take into account time?


    A post hoc sample size / power calculation could be frown upon, if it directs you to collect more data to just try and get significance.




    If variables are truly independent and you are not looking to increase sample size or kept the study going, you could easily get an idea of power issues by just calculating it for bivariate fisher test power analyses (if IVs are categorical), since you have a finite sample size. So run a power analysis for Fisher exact test for a outcome and variable.


    Another option would be to run simulations to determine what sample size would be need for distinguishing say an AUC value of 0.55 from 0.50.
    Stop cowardice, ban guns!

  3. #3
    Points: 3,490, Level: 36
    Level completed: 94%, Points required for next Level: 10

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Statistical significance

    Quote Originally Posted by hlsmith View Post
    Did you use multiple logistic regression to do this?


    Most people will say that with only 11 events, your sample size may only support 1 predictor at most. How many predictors did you examine and how were they formatted (e.g., binary variables), and did you test them one at a time? A p-value > 0.5 for an AUC value just means the variable is not any better then chance, and since they were close to 0.5, that doesn't bode well for it just being a small sample size thing.


    Since your outcome was death status, was it not feasible to examine data with survival model which would take into account time?


    A post hoc sample size / power calculation could be frown upon, if it directs you to collect more data to just try and get significance.




    If variables are truly independent and you are not looking to increase sample size or kept the study going, you could easily get an idea of power issues by just calculating it for bivariate fisher test power analyses (if IVs are categorical), since you have a finite sample size. So run a power analysis for Fisher exact test for a outcome and variable.


    Another option would be to run simulations to determine what sample size would be need for distinguishing say an AUC value of 0.55 from 0.50.


    Hi,

    Thank you very much for your answer!

    I am interested in assessing whether or not a certain variable is able to predict the outcome. Even a negative result (i.e. the variable cannot predict the outcome) is an interesting result for me. However, I would like to be sure that the negative result I obtain is not merely due to the small sample size.
    I tested each variable separately and used a logistic regression (not a multiple one, since it would not make so much sense for my problem to combine the variables together in the same model).
    The variables I tested were 8, all of them were continuous variables.

    I have the day of death of these patients and I could perform a survival analysis, but I am interested in knowing whether or not the variable under consideration is able to predict the outcome and, in the positive case (statistical significance), I would like to know the threshold for this variable having the highest sensitivity and specificity (optimal criterion). This is the reason why I decided to perform an ROC analysis.
    Do you think that a survival analysis would be a better option?

    Unfortunately, the number of patients cannot change; I am not able to collect more data.
    I am not familiar at all with sample size calculations and power issues, but I tried to run a test for the sample size and AUC values in the software I use (MedCalc). My results are in the attachment. I am not sure if I performed the test correctly. Results seem to indicate that to differentiate between an AUC value of 0.55 and 0.5 I would need more than 600 patients (and this is not doable for me). What is your opinion?
    Thank you very much again!
    Attached Images  

  4. #4
    TS Contributor
    Points: 19,199, Level: 87
    Level completed: 70%, Points required for next Level: 151
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,080
    Thanks
    123
    Thanked 429 Times in 330 Posts

    Re: Statistical significance

    Looks like your messages were being flagged by our spam software (likely due to the use of attachments, given that you're a new-ish user). I've released your most recent post and deleted the two duplicates. Sorry about the delay.
    Matt aka CB | twitter.com/matthewmatix

  5. #5
    Omega Contributor
    Points: 39,128, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,085
    Thanks
    402
    Thanked 1,194 Times in 1,155 Posts

    Re: Statistical significance

    What program did you use for the regression?

    So did you dichotomize the continuous variable and that is where the 0.55 is coming from?

    Just curious what your SEN, SPEC values are for the variable.

    If you can post your 2x2 confusion matrix I may play around with your numbers. Surprisingly I don't run many survival curves, but given a linear relationship between the continuous variable and outcome, you could get probabilities for values of continuous var for the outcome and hazard ratios which are like relative risks. Depends on what your question is???
    Stop cowardice, ban guns!

  6. #6
    Points: 3,490, Level: 36
    Level completed: 94%, Points required for next Level: 10

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Statistical significance

    Hi hlsmith,
    I posted my reply here with all my calculations attached a few days ago but it was stopped again. I tried to contact both CowboyBear directly and the administrators multiple times in order to fix this problem but I haven't got any reply yet. I hope they will make my post visible soon.
    Thank you for your help!

  7. #7
    TS Contributor
    Points: 19,199, Level: 87
    Level completed: 70%, Points required for next Level: 151
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,080
    Thanks
    123
    Thanked 429 Times in 330 Posts

    Re: Statistical significance

    There's no post showing here waiting to be released - whatever the problem is may be something other than the spam filter?
    Matt aka CB | twitter.com/matthewmatix

  8. #8
    Points: 3,490, Level: 36
    Level completed: 94%, Points required for next Level: 10

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Statistical significance

    Ok, thank you for your reply! Then I will try to post it again!

  9. #9
    Points: 18, Level: 1
    Level completed: 35%, Points required for next Level: 32

    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Statistical significance


    Thanks for the insightful response

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats