Time series regression with blocks of years?

I am evaluating the impact of policy changes over time. There are three separate time periods, each representing a number of years, and I want to see how the policy in place during each block effects a number of variables. How do I set that up? Is there a better option than what I'm thinking?


Fortran must die
Well you can run ARIMA with an intervention indicator when the policy changes as suggested by Hays et el, one for the impact on each dependent variable possibly (I assume here the variables you reference are being influenced by the change not causing the change). This won't tell you, statistically, about the difference between each period (each block). There is the (I think this is the correct test) a Chow test of structural breaks over time which you might look at and see if it meets your need. It looks for structural breaks in time series data.

Why are you breaking the data up into 3 periods. Normally you want at least 50 points of data for time series, so breaking the data up this way seems less than ideal.
Thanks for the suggestions. The policy changed twice, so it naturally breaks up into three sets of years if I look at each separately. The data is by year though, which is only 20 data points, so I'm still not sure time series is the best option. Not a statistician, and my PhD supervisors are qualitative researchers and don't know anything about statistics, so I'm kind of just searching for possible methods at the moment.


Fortran must die
A number of well known methods (such as the ARIMA intervention approach I mentioned) are not ideal with so few data points. It is hard to estimate seasonality for one thing, and I believe the power of the test would be low with so few points. The common recommendation is at least 50 (and some commentators suggest well more than that).

I think you could create a dummy variable (one for each intervention) taking on the value of 0 before the intervention and 1 after it and run this in regression. If the intervention had an impact the slope of the dummy should be signficant. But it has been a while since I read that literature - you might want to start with that literature. You would likely have problems with power with only 20 data points. That is there could be an impact and you would not capture it because you had too little data (formally a type II error).

You would also have problems if the impact was gradual rather than immediate or if it declined quickly (or worse gradually) after the intervention returning to preintervention levels. You can read David McDowall et el "Interrupted Time Series Analysis" Sage 1980 to get a feel for these issues. They deal with it through ARIMA (which you probably could not use), but I would guess the issues are the same. One possibility you could consider is Interrupted Time Series without any formal statistical test such as regression. That is literally look at the mean level of the dependent variables before and after the intervention. There would not be a formal statistical test this way, however.

The relationship you find before and after the intervention can always be spurious (other factors actually caused the change that coincided with the intervention). One solution for this is to look at say a different state or county that did not experience the intervention (depending on your unit of analysis) and see what the result is after the intervention. If the intervention unit did experience a change and the unit which did not experience had no change it provides more support for the impact of the intervention. Obviously you want a unit similar except for if it experienced the intervention.

I would look at interrupted time series in general. It probably is how you want to go. You have my sympathy on your comittee. I did a qualitative disertation for a quantitative group with small love for qualitative methods - and it was no fun. Best wishes....:)