pre- post- intervention survey analysis

S

siny911

New Member
#1
#1
Hi all,

I am trying to figure out how to analyze survey results administered as part of a randomized trial. We drew random samples of households from 4 different locations: 1 neighborhood which received treatment a, 1 neighborhood which received treatment b, 1 neighborhood that received treatment c, and finally one neighborhood that received no treatment and acted as a control. This was a non-panel survey administered prior to the treatment period and following the treatment period. In other words, the same sample of households were mailed the surveys before and after the intervention, but we cannot match respondents across waves, so we basically have 2 crosssections of respondents. Therefore I cannot run a repeated measures anova. I was thinking of running an anova with a variable distinguishing between the treatment type, the time the survey was administered (pre or post), and then the interaction between the type of treatment and the time of survey. Am I way off here? Any suggestions would be appreciated.
 
S

siny911

New Member
#2
#2
my dependent variable will be a scale (interval/ratio) based on responses to several questions in the survey which captures 'satisfaction with town services'
 
noetsi

noetsi

No cake for spunky
#3
#3
If you sent the surveys to the same households, why can't you match respondents? Because you don't know if the same respondents lived in the same houses?

If the people who are measured in your ANOVA before the intervention are not the same people after the intervention then I would think the only way you could use ANOVA would be if you assume only neighborhood and intervention effects matter not individual ones (so which person answered the survey did not matter). In that case I would think you could run repeated measures.

If indivudal differences matter than it would seem you have a major confound that ANOVA could not address. That is any finding you get might reflect changes in individual differences of respondents before and after the intervention rather than intervention effects.

For example the number of years someone lived in the neighborhood might be very different in the pre-intervention and post intervention and that (rather than intervention) might explain the results.

But then I am not an expert in DOE or ANOVA so I could be wrong.
 
S

siny911

New Member
#4
#4
I can't match the respondents because IRB did not allow us to collect identifiable information because of the nature of the intervention and where the surveys were distributed. So I can't do a repeated measures ANOVA. I essentially have a repeated cross-sectional survey. What I was thinking is that I would run a factorial ANOVA, where the dependent variable is an interval level additive scale based on responses to survey questions which measure the construct 'satisfaction with town services'. Then I would have 2 independent variables. 1 would be the time period the survey was administered (1= before intervention, 2=after intervention). Then a treatment type variable (1= treatment a, 2=treatment b, 3=treatment c, 4= control). I could then interact the treatment and time variable. If there is a significant interaction I could perform post-hoc tests to see if there are difference from before to after the treatment was administered for any of the treatments relative to the control condition. Does this make sense/anyone see any issues with this?
 
noetsi

noetsi

No cake for spunky
#5
#5
To me the issue is what I noted previously. How to insure that changes in the respondents before and after intervention did not cause the results you discover. If you can't do this I dont see how you can be certain that the effects you noted cause the results you find.
 
S

siny911

New Member
#6
#6
I agree that this will be a limitation, but given this is an RCT, I am operating under the assumption that any of those effects would be equal across neighborhoods/treatment type.
 
Lazar

Lazar

Phineas Packard
#7
#7
Here is my two cents:
1. In typically situations, with random assignment your groups should be balanced on all covariates so you can take the effect of group at post-treatment to represent the average treatment effect BUT the problem is that your random assignment is at the neighborhood level not the participant level so effectively you have an n of 4 in relation to random assignment. I think it is thus an unreasonable assumption to think that on average your four groups are balanced on all covariates.
2. Your best bet is to treat this as a quasi-experimental research. In this case I would be using the pre-treatment data and check for balance in variables of interest and as many covariates as possible. If you are lucky then there will be balance and you therefore have somewhat stronger evidence to say that the post-treatment effect is a treatment effect.
 
Lazar

Lazar

Phineas Packard
#8
#8
siny911 said:
I agree that this will be a limitation, but given this is an RCT, I am operating under the assumption that any of those effects would be equal across neighborhoods/treatment type.
Click to expand...
Yes but the RCT only has an n of 4 because you randomly assigned at the neighborhood level not at the participant level. I think it is too strong an assumption to trust random assignment with such a small N. In particular if there are any neighborhood effects then your random assignment will not work to balance on all covariates.
 
S

siny911

New Member
#10
#10
Lazar said:
Yes but the RCT only has an n of 4 because you randomly assigned at the neighborhood level not at the participant level. I think it is too strong an assumption to trust random assignment with such a small N. In particular if there are any neighborhood effects then your random assignment will not work to balance on all covariates.
Click to expand...
Thinking about thus further... I actually randomized on n=120 neighborhoods, but 4 treatment types. I'm still thinking I should treat this as a quasi-experiment though. Thoughts?
 
Lazar

Lazar

Phineas Packard
#11
#11
Hang on you randomly assigned 120 neighborhoods to four groups? If it was true random assignment then I think you are fine (if you do your analysis at the neighborhood level then there will be no problem, I am not sure of the stable treatment unit value assumption implications of conducting analysis at the individual level BUT my guess is that multilevel ANOVA with individuals nested within neighborhoods would help deal with this).

I think I can give better advice if I know whether the unit of analysis is the individual or the neighborhood. What are you trying to effect a change in?

EDIT: STUVA is not an issue here. Still knowing the unit of analysis is important.
 
You must log in or register to reply here.