+ Reply to Thread
Results 1 to 9 of 9

Thread: External Validation of Cox Model

  1. #1
    Points: 51, Level: 1
    Level completed: 2%, Points required for next Level: 49

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    External Validation of Cox Model




    I have two data set (training and validation) for building and validating a Cox model.

    With the training data set I fitted a cox model using stepwise selection method.

    The significant variables in the model were the only variables included in the validation model. Is this the right approach?

    While validating the model I realized that the variables are not significant in the validation model and also the assumptions of the cox model do not hold (I checked the assumption on the validation data). Should I ignore the fact that the variables are insignificant and go ahead in making corrections for the problem with model assumptions in validation data?

    Thirdly,in both training and validation data I have a variable 'treatment' with three groups. In training the groups are Standard, New drug and mixture, while in validation data the groups are Standard, New drug and X (is a treatment which is different from mixture in training data). Is it right to include this variable in both model or should I eliminate the groups that are not match; mixture from training data and X from validation data or should I work with it like that? I am not sure how this affects my analysis.

    Thanks for your responses.

  2. #2
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: External Validation of Cox Model

    I am guessing the you did not randomly split the data into a training and validation set?
    Stop cowardice, ban guns!

  3. #3
    Points: 51, Level: 1
    Level completed: 2%, Points required for next Level: 49

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: External Validation of Cox Model

    No I did not. The data were collected from two different experiment.

  4. #4
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: External Validation of Cox Model

    Systematic bias (selection bias, measurement bias, and possibly confounding) along with chance is gonna mess with you. Yeah, you usually take the coefficients from the test set and score the validation set and see how accurate the model is in the validation set. So you wont rerun the model. I don't use PHREG regularly, so I can't remember the assumptions or if you would care too much about them given the above approach.


    What assumptions seem to be broken?
    Stop cowardice, ban guns!

  5. #5
    Points: 51, Level: 1
    Level completed: 2%, Points required for next Level: 49

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: External Validation of Cox Model

    Quote Originally Posted by hlsmith View Post
    Systematic bias (selection bias, measurement bias, and possibly confounding) along with chance is gonna mess with you. Yeah, you usually take the coefficients from the test set and score the validation set and see how accurate the model is in the validation set. So you wont rerun the model. I don't use PHREG regularly, so I can't remember the assumptions or if you would care too much about them given the above approach.


    What assumptions seem to be broken?

    Proportionality. Though I can stratify the affected variable but then I would loss information on it. I could introduce an interaction but then I don't want to complicate my interpretation.

  6. #6
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: External Validation of Cox Model

    Hmm. I have not cross-validated for PHREG. You could go back and make this change to the original model as well, but then you are letting validation influence testing and it may not even effect the original model, but to take up degrees of freedom.


    I think it just comes back to the differences in the datasets. This is an interesting problem, though I don't know enough to move you forward. Can you explain the differences between the datasets or anything that may make them unique. Is it possible to merge them or use a shrinkage approach. However, these may not help you out given the problem.
    Stop cowardice, ban guns!

  7. #7
    Points: 51, Level: 1
    Level completed: 2%, Points required for next Level: 49

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: External Validation of Cox Model

    Quote Originally Posted by hlsmith View Post
    Hmm. I have not cross-validated for PHREG. You could go back and make this change to the original model as well, but then you are letting validation influence testing and it may not even effect the original model, but to take up degrees of freedom.


    I think it just comes back to the differences in the datasets. This is an interesting problem, though I don't know enough to move you forward. Can you explain the differences between the datasets or anything that may make them unique. Is it possible to merge them or use a shrinkage approach. However, these may not help you out given the problem.
    That was my thought (letting validation influencing testing)!

    The only difference about the data set is that one of the treatment arm in validation data set contains a different treatment from we have in testing data set. Though out of curiosity I decided to remove observations from the different arms and used treatment arms that were common to both data set still it couldn't fix the problem.

  8. #8
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: External Validation of Cox Model

    So test dataset has 2 treatments and validation has 3 treatments? Were treatments randomized?
    Stop cowardice, ban guns!

  9. #9
    Points: 51, Level: 1
    Level completed: 2%, Points required for next Level: 49

    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: External Validation of Cox Model


    Quote Originally Posted by hlsmith View Post
    So test dataset has 2 treatments and validation has 3 treatments? Were treatments randomized?
    test dataset has 3 treatment as well. The only difference is that the third treatment differs in both dataset.

    Yes treatment was randomized.

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats