pre-test post-test difficulty

[rclukey]

Hi everyone,
I'm new to the forum, but I'm definitely glad it's here.

I have a difficult research design problem / stats problem that I was hoping to get some advice on.

I have a situation where we are trying to determine not only what the effect of a treatment is before and after, but what aspects or factors of that treatment affect the post-test measurement. in this case it is a web based learning training tool; users don't have to utilize all aspects of the training tool to complete the training, so we'd like to understand what of the aspects they do use have the most impact on the training outcome

I have read some about separate sample pre-test post-test designs, as well as repeated measures. However, the part I'm struggling with is how to measure the specific tool features utilized by our sample and how to place them within a standard pre-test post-test design to help predict the DV.

If we use a repeated measures design, then I can use the pre-test as a covariate.

here are some basics:
measurements are being done using an online survey tool
sample is randomly selected, we cannot control who partakes in the training / use of tool
we can track who uses what features of the tool

Any help or advice is appreciated.
thank you kindly.

 

[JohnM]

This sounds like more of a correlational study than a designed study, so you may be limited to just looking at apparent associations rather than concluding cause-and-effect....at least for now. If I read correctly, the participants are free to do what they want, so there are a lack of controls that would be expected in a designed, controlled experiment.

Basically, for each participant, you can compare pre- and post-test scores and get a general sense of whether, from an overall sense, the training tool was effective.

In terms of what aspects of the training were most important, you'll need to examine the activity data to see if any correlations can be drawn between difference scores (post-treatment minus pre-treatment) and participation in certain activities.

Once you've identified the most important or strongest apparent correlations, you should then design a controlled study to get a better understanding of exactly how important they are and confirm whether the difference is statistically significant.