+ Reply to Thread
Results 1 to 14 of 14

Thread: pre- post- intervention survey analysis

  1. #1
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    pre- post- intervention survey analysis




    Hi all,

    I am trying to figure out how to analyze survey results administered as part of a randomized trial. We drew random samples of households from 4 different locations: 1 neighborhood which received treatment a, 1 neighborhood which received treatment b, 1 neighborhood that received treatment c, and finally one neighborhood that received no treatment and acted as a control. This was a non-panel survey administered prior to the treatment period and following the treatment period. In other words, the same sample of households were mailed the surveys before and after the intervention, but we cannot match respondents across waves, so we basically have 2 crosssections of respondents. Therefore I cannot run a repeated measures anova. I was thinking of running an anova with a variable distinguishing between the treatment type, the time the survey was administered (pre or post), and then the interaction between the type of treatment and the time of survey. Am I way off here? Any suggestions would be appreciated.

  2. #2
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: pre- post- intervention survey analysis

    my dependent variable will be a scale (interval/ratio) based on responses to several questions in the survey which captures 'satisfaction with town services'

  3. #3
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: pre- post- intervention survey analysis

    If you sent the surveys to the same households, why can't you match respondents? Because you don't know if the same respondents lived in the same houses?

    If the people who are measured in your ANOVA before the intervention are not the same people after the intervention then I would think the only way you could use ANOVA would be if you assume only neighborhood and intervention effects matter not individual ones (so which person answered the survey did not matter). In that case I would think you could run repeated measures.

    If indivudal differences matter than it would seem you have a major confound that ANOVA could not address. That is any finding you get might reflect changes in individual differences of respondents before and after the intervention rather than intervention effects.

    For example the number of years someone lived in the neighborhood might be very different in the pre-intervention and post intervention and that (rather than intervention) might explain the results.

    But then I am not an expert in DOE or ANOVA so I could be wrong.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  4. #4
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: pre- post- intervention survey analysis

    I can't match the respondents because IRB did not allow us to collect identifiable information because of the nature of the intervention and where the surveys were distributed. So I can't do a repeated measures ANOVA. I essentially have a repeated cross-sectional survey. What I was thinking is that I would run a factorial ANOVA, where the dependent variable is an interval level additive scale based on responses to survey questions which measure the construct 'satisfaction with town services'. Then I would have 2 independent variables. 1 would be the time period the survey was administered (1= before intervention, 2=after intervention). Then a treatment type variable (1= treatment a, 2=treatment b, 3=treatment c, 4= control). I could then interact the treatment and time variable. If there is a significant interaction I could perform post-hoc tests to see if there are difference from before to after the treatment was administered for any of the treatments relative to the control condition. Does this make sense/anyone see any issues with this?

  5. #5
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: pre- post- intervention survey analysis

    To me the issue is what I noted previously. How to insure that changes in the respondents before and after intervention did not cause the results you discover. If you can't do this I dont see how you can be certain that the effects you noted cause the results you find.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  6. #6
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: pre- post- intervention survey analysis

    I agree that this will be a limitation, but given this is an RCT, I am operating under the assumption that any of those effects would be equal across neighborhoods/treatment type.

  7. #7
    Phineas Packard
    Points: 16,013, Level: 81
    Level completed: 33%, Points required for next Level: 337
    Lazar's Avatar
    Location
    Sydney
    Posts
    1,159
    Thanks
    198
    Thanked 336 Times in 299 Posts

    Re: pre- post- intervention survey analysis

    Here is my two cents:
    1. In typically situations, with random assignment your groups should be balanced on all covariates so you can take the effect of group at post-treatment to represent the average treatment effect BUT the problem is that your random assignment is at the neighborhood level not the participant level so effectively you have an n of 4 in relation to random assignment. I think it is thus an unreasonable assumption to think that on average your four groups are balanced on all covariates.
    2. Your best bet is to treat this as a quasi-experimental research. In this case I would be using the pre-treatment data and check for balance in variables of interest and as many covariates as possible. If you are lucky then there will be balance and you therefore have somewhat stronger evidence to say that the post-treatment effect is a treatment effect.

  8. #8
    Phineas Packard
    Points: 16,013, Level: 81
    Level completed: 33%, Points required for next Level: 337
    Lazar's Avatar
    Location
    Sydney
    Posts
    1,159
    Thanks
    198
    Thanked 336 Times in 299 Posts

    Re: pre- post- intervention survey analysis

    Quote Originally Posted by siny911 View Post
    I agree that this will be a limitation, but given this is an RCT, I am operating under the assumption that any of those effects would be equal across neighborhoods/treatment type.
    Yes but the RCT only has an n of 4 because you randomly assigned at the neighborhood level not at the participant level. I think it is too strong an assumption to trust random assignment with such a small N. In particular if there are any neighborhood effects then your random assignment will not work to balance on all covariates.

  9. #9
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: pre- post- intervention survey analysis

    Excellent point and good idea. Thanks!

  10. #10
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: pre- post- intervention survey analysis

    Quote Originally Posted by Lazar View Post
    Yes but the RCT only has an n of 4 because you randomly assigned at the neighborhood level not at the participant level. I think it is too strong an assumption to trust random assignment with such a small N. In particular if there are any neighborhood effects then your random assignment will not work to balance on all covariates.
    Thinking about thus further... I actually randomized on n=120 neighborhoods, but 4 treatment types. I'm still thinking I should treat this as a quasi-experiment though. Thoughts?

  11. #11
    Phineas Packard
    Points: 16,013, Level: 81
    Level completed: 33%, Points required for next Level: 337
    Lazar's Avatar
    Location
    Sydney
    Posts
    1,159
    Thanks
    198
    Thanked 336 Times in 299 Posts

    Re: pre- post- intervention survey analysis

    Hang on you randomly assigned 120 neighborhoods to four groups? If it was true random assignment then I think you are fine (if you do your analysis at the neighborhood level then there will be no problem, I am not sure of the stable treatment unit value assumption implications of conducting analysis at the individual level BUT my guess is that multilevel ANOVA with individuals nested within neighborhoods would help deal with this).

    I think I can give better advice if I know whether the unit of analysis is the individual or the neighborhood. What are you trying to effect a change in?

    EDIT: STUVA is not an issue here. Still knowing the unit of analysis is important.

  12. #12
    Points: 994, Level: 16
    Level completed: 94%, Points required for next Level: 6

    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: pre- post- intervention survey analysis

    I sent you a private message Lazar

  13. #13
    Devorador de queso
    Points: 95,754, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,932
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: pre- post- intervention survey analysis

    What did you send that you couldn't post here?
    I don't have emotions and sometimes that makes me very sad.

  14. #14
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: pre- post- intervention survey analysis


    Plans to eliminate all raptors with a genetically altered compound
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  15. The Following User Says Thank You to noetsi For This Useful Post:

    Dason (03-05-2013)

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats