+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 16

Thread: comment on research design.

  1. #1
    Fortran must die
    Points: 62,031, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,538
    Thanks
    693
    Thanked 916 Times in 875 Posts

    comment on research design.




    I have been asked to create a research design. The goal is to evaluate a new way of serving customers, an intervention to do so, with a series of goals. For each goal (which can be measured quantitatively), I have proposed an interrupted time series comparing results in the previous 48 months to progress after the intervention with time measured on a monthly interval.

    The change in levels and trend will suggest if the intervention is successful although as with all such approaches other factors that occur at the same time might drive the change not the intervention. I suggested they add a control by comparing a unit this way that did not have an intervention to a unit that was similar that did have an intervention (it remains to be seen if we can find such units).

    Finally I suggested a regression to, distinct from the time series, look at whether the intervention had an impact (you just add an intervention variable that is 1 in the intervention period and the future and zero before it).

    I am sure there are many flaws in this approach, I would appreciate posters suggesting improvements.
    Attached Images  
    Last edited by noetsi; 10-31-2017 at 06:08 PM.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  2. #2
    Human
    Points: 12,955, Level: 74
    Level completed: 27%, Points required for next Level: 295
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,401
    Thanks
    460
    Thanked 474 Times in 414 Posts

    Re: comment on research design.

    I would suggest a factorial design with many factors. (Doing just on-factor-at-a-time is so boring and bad.)

    Randomize a small sample (but large enough to get power) out of the population. Make sure there is a control group. Make sure that each unit (customer?) will get the decided treatment.

    What is the most important response variable?

  3. The Following User Says Thank You to GretaGarbo For This Useful Post:

    noetsi (10-31-2017)

  4. #3
    Omega Contributor
    Points: 39,045, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,070
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: comment on research design.

    No that is a good design. I did an interrupted time series last year, but wasn't able to have a control group.

    Can you randomize who gets the change? If so, that helps clear up imbalances between groups (control vs intervention).
    Stop cowardice, ban guns!

  5. #4
    Fortran must die
    Points: 62,031, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,538
    Thanks
    693
    Thanked 916 Times in 875 Posts

    Re: comment on research design.

    I appended the indicators (and the metrics which will measure it) they want to my original post.

    I am not sure I have the expertise to do a multifactorial design. I have worked with regression a lot but not ANOVA. But I am not sure I understand what you mean by this

    I am not sure how to do a control group, other than what I suggested. Everyone in the unit will have the intervention. The only way to have a control is to compare one unit to another.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  6. #5
    Fortran must die
    Points: 62,031, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,538
    Thanks
    693
    Thanked 916 Times in 875 Posts

    Re: comment on research design.

    Quote Originally Posted by hlsmith View Post
    No that is a good design. I did an interrupted time series last year, but wasn't able to have a control group.

    Can you randomize who gets the change? If so, that helps clear up imbalances between groups (control vs intervention).
    No everyone in the unit will get the change. It is a process change to move customers through the organization more effectively. There is no option for some to get the intervention and some not, but different units will start the intervention at different times.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  7. #6
    Omega Contributor
    Points: 39,045, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,070
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: comment on research design.

    Economists love these designs, and you are talking about rolling thing out to the population level so its a good fit.

    I am also unsure of the design Greta is specifically recommending!
    Stop cowardice, ban guns!

  8. The Following User Says Thank You to hlsmith For This Useful Post:

    GretaGarbo (10-31-2017)

  9. #7
    Human
    Points: 12,955, Level: 74
    Level completed: 27%, Points required for next Level: 295
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,401
    Thanks
    460
    Thanked 474 Times in 414 Posts

    Re: comment on research design.

    Quote Originally Posted by noetsi View Post
    No everyone in the unit will get the change.
    What is "a unit"?

  10. #8
    Omega Contributor
    Points: 39,045, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,070
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: comment on research design.

    I am guessing a geographical area (e.g., county)?
    Stop cowardice, ban guns!

  11. #9
    Human
    Points: 12,955, Level: 74
    Level completed: 27%, Points required for next Level: 295
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,401
    Thanks
    460
    Thanked 474 Times in 414 Posts

    Re: comment on research design.

    Quote Originally Posted by hlsmith View Post
    I am guessing a geographical area (e.g., county)?
    Then randomize to these units then.

    Noetsi, what is a unit?

    I can't read noetsi's attachment in post 1.

  12. #10
    Fortran must die
    Points: 62,031, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,538
    Thanks
    693
    Thanked 916 Times in 875 Posts

    Re: comment on research design.

    We provide service to customers through units (which are rolled up into larger districts run by directors) which serve geographical areas, although some units overlap. I don't think I can randomize them because all units in each district will get the intervention at the same time. I can chose similar units in areas that do get the intervention and those that don't (everyone will get the intervention, but it will roll out to areas by months).

    What is better. Randomize units that do and do not get the intervention, or select units that do or do not get the intervention based on similarity on dimensions thought to be important.

    I will reattach the document. Not sure this will be any better.
    Attached Images  
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  13. #11
    Omega Contributor
    Points: 39,045, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,070
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: comment on research design.

    Ideally you have a before and after time period and another group naive to intervention neither before or after. Having gradual roll out also does this, given there aren't big season effects.
    Stop cowardice, ban guns!

  14. #12
    Human
    Points: 12,955, Level: 74
    Level completed: 27%, Points required for next Level: 295
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,401
    Thanks
    460
    Thanked 474 Times in 414 Posts

    Re: comment on research design.

    Quote Originally Posted by hlsmith View Post
    Ideally you have a before and after time period and another group naive to intervention neither before or after. Having gradual roll out also does this, given there aren't big season effects.
    I believe that hlsmith mean that there must be a control group. Otherwise it would be meaningless.

    Suppose I say: "let the weather be colder" and I observe colder weather. Does that mean that I can control the weather? Or simply that the winter is coming, so that there is an underlying time trend.

    Maybe noetsi want to give us examples of what treatments that can be changed and what to be measured (maybe in the lounge)?

  15. #13
    Omega Contributor
    Points: 39,045, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,070
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: comment on research design.

    Well the Interrupted Time Series is a more sophisticated before-after study where you take into account the change in level and trend. So you have a before slope and a after slope and possibly a level change, which is better than the basic before and after design. Now your before is a comparison group as long as you have a sufficient amount of time data to truly understand seasonality and stochashic-ness.


    Now in addition to having this self-control group, you can also have a control group that doesn't get the intervention or in your case gets it at a delayed time. This adds an addition amount of supporting evidence. Though the one thing you may need to watch out for is this could also be a multi-level model if you have observations clusters in regions. There is a chance that you can just control for this in normal 1-level regression instead of controlling for the second level. There are general rules for the amount of time before and after the change that needs to be collected as well as the amount of observations per time point, but these are usually just related to having a representative sample across time.


    I am excited for you Noetsi, hope this works out well for you.
    Stop cowardice, ban guns!

  16. #14
    Fortran must die
    Points: 62,031, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,538
    Thanks
    693
    Thanked 916 Times in 875 Posts

    Re: comment on research design.

    I will give an example of what I mean. One of the goals for the imitative is to increase the number of applicants relative to referrals (a ratio). I was going to look at the ratio for the last 48 months for a geographic unit (or possibly units) before the imitative and then at least the next 12 months after the initiative. I was going to compare this to a "control" (formally I know this is not a control) of a unit that did not get the intervention (for the same time period). Ideally it will be in the same area and as close as I can find to similar to the one getting the intervention although it might have to be in a different area not the one with the intervention.

    There may be differences between areas that influences the result although we do not know this. One problem is that no one really knows what drives these factors (the literature in vocational administration is not sophisticated statistically IMHO and no one here has thought much on this topic). So controlling for factors other than the intervention, be that through statistical controls in regression or matching a control unit to a non-control one is difficult.

    Greta Garbo I would like to discuss this in much more detail possibly in the lounge but for the next few weeks I am swamped by a major rewrite of SQL required by the federal government (structured query language for our report).
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  17. #15
    Omega Contributor
    Points: 39,045, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,070
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: comment on research design.


    If there are differences between the groups that may effect the outcome you can control for them directly or with inverse propensity scores in your interrupted time series. You will need to also think about if your model will by linear or beta regression, or something else.
    Stop cowardice, ban guns!

+ Reply to Thread
Page 1 of 2 1 2 LastLast

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats