+ Reply to Thread
Results 1 to 8 of 8

Thread: Combining Two Samples (one w/ intervention and one w/ out intervention)

  1. #1
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Combining Two Samples (one w/ intervention and one w/ out intervention)




    I have a local dataset with a continuous predictor and binary outcome. I will use logistic regression to predict the binary outcome with the continuous predictor.


    I also have a national sample of comparable individuals, along with the same continuous predictor and binary outcome. I will duplicate the above approach. Though all individuals in the second dataset also received an intervention after the collection of the continuous variable and before the outcome.


    I am debating about whether it is feasible to also merge these two datasets together and predictor the binary outcome with the continuous variable while controlling for and testing the effect of the intervention status?


    I wonder if I can do this or what I need to think about. I would assume (and also test) that the only difference between the datasets is that the national group got an intervention and background characteristics are comparable between datasets. Then examine the effect of the intervention on the outcome.


    I am feeling that I may need to run a multi-level model. Then look to see if dataset of origin was a significant predictor (AKA intervention status). Though, I will only have two datasets of origin (local and national), so two groups in level two of the model.


    So taken home question, can I merge these sets together and test the predictive value of the continuous variable as well as effect for which dataset data came from using a multi-level model?


    Any thoughts?
    Stop cowardice, ban guns!

  2. #2
    TS Contributor
    Points: 18,889, Level: 87
    Level completed: 8%, Points required for next Level: 461
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,062
    Thanks
    121
    Thanked 427 Times in 328 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)

    You could merge the two. One thing to think about is that, since everyone in dataset2 got the intervention, the two variables "intervention vs no intervention" and "sample1 vs sample2" are perfectly correlated. Thus making it very hard to work out what is the effect of the intervention vs. what pre-existing differences there might be across the two samples. As a study of the effect of the intervention, what you have is in effect a static group comparison (a pre-experimental design), so your inferences will be very tentative. On the other hand, this doesn't mean you can't get a good assessment of the relationship between the continuous predictor and the binary outcome.

    I would just want to think about what other differences there might be between the two datasets (in demographics, measurement procedures, etc).

    I can't quite envisage how a multilevel model would apply in this instance, but feel free to talk more about your ideas on that.

  3. #3
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)

    I agree with CB.
    Quote Originally Posted by hlsmith View Post
    I would assume (and also test) that the only difference between the datasets is that the national group got an intervention and background characteristics are comparable between datasets.
    That's a big assumption, and how exactly do you plan on testing it? It looks to me as if you have no means of testing it.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  4. #4
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)

    Thanks for feedback.

    I neglected to out and out say it, but I have other variables, which could be used to examine background charecteristics.

    The reason I was thinking multilevel model was the use of them in meta -analyses to control for heterogeneity and to see if cluster (dataset) level data can account for additional model covariance.

    I get that assumptions need to be made that are not testable, say related to "s-admissibility" since unknown/not collected confounders could exist.

    I don't get why a multilevel model wouldn't work. Say you are looking at IQ in predicting a binary variable and you want to control for which school they went to. Are you saying that just including a varaiable school A or B would be better than a multilevel model, why? I get this is not an ideal set up overall.
    Stop cowardice, ban guns!

  5. #5
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)

    Quote Originally Posted by hlsmith View Post
    Are you saying that just including a varaiable school A or B would be better than a multilevel model, why? I get this is not an ideal set up overall.
    Yes. You can't really fit a multilevel model with only two clusters. In a multilevel model you estimate the between-cluster variance, but getting a variance estimate based on only 2 data points is difficult to say the least. You can try to fit the model but you will probably run into convergence errors. It's generally recommended that you have something like 10 or more clusters.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  6. The Following User Says Thank You to Jake For This Useful Post:

    hlsmith (05-02-2016)

  7. #6
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)

    Yes that was one of my initial concerns. I need to read a book on ML. What do you think of gelmans book or do you have a recommendation for a novice? I think besides convergence, that the SE contribution would be very large.
    Stop cowardice, ban guns!

  8. #7
    TS Contributor
    Points: 18,889, Level: 87
    Level completed: 8%, Points required for next Level: 461
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,062
    Thanks
    121
    Thanked 427 Times in 328 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)

    +1 for Gelman's book.

  9. #8
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Combining Two Samples (one w/ intervention and one w/ out intervention)


    Gelman & Hill is good. I also like Pinheiro & Bates and Snijders & Bosker.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats