Thread: Is survey design statistically correct?

1. Is survey design statistically correct?

I want to track improvements in how yoga class participants feel on various variables (e.g. energy level, sleep quality, pain felt, mental relaxation) before and after a four-day intensive training course.

The design is to give each participant a questionnaire at the start of the course where they answer how they feel currently about each tracked variable, for example their energy level, sleep quality, extent of pain, level of mental relaxation, on a scale of 1 to 10.

Then, after the course, the same participants will answer the same questions again, to evaluate whether they improved on any variable.
So, the answers are not anonymous, and we can know who answered what at the start as well as the end of the course. We are confident that there will be no problem with the response rate in spite of the answers not being anonymous.

Is my study design statistically correct? Or how could it be improved?

And after collecting the data, should I do the paired t-test to check the statistical significance of the differences at the start and after the course for each tracked variable? Is this the right test?

Thanks for any help!

2. Re: Is survey design statistically correct?

The design is to give each participant a questionnaire at the start of the course where they answer how they feel currently about each tracked variable, for example their energy level, sleep quality, extent of pain, level of mental relaxation, on a scale of 1 to 10.
Is there any possibility to create a control condition, e.g. a waiting-list control group? Or, something like a "placebo" condition, i.e. a course which somehow resembles the yoga course but hasn't the crucial yoga elements included.
We are confident that there will be no problem with the response rate in spite of the answers not being anonymous.
You maybe could ask participants to create a code for themselves (something like "first 2 letters of mother's forename + first 2 letters of father's forename + last 2 digits of postal code of the location where one was born").
Is my study design statistically correct? Or how could it be improved?
You have no control group, therefore you cannot estimate whether pre-post differences are causally related to attending the yoga class.

And after collecting the data, should I do the paired t-test to check the statistical significance of the differences at the start and after the course for each tracked variable? Is this the right test?
Seems so. But you did not mention how large your sample size is. And, is this just 1 yoga class, or are several classes involved?
And you'd usually need to survey some additional characteristics, in order to describe the sample
(age, gender, maybe pre-experience with yoga and the like...).

Just my 2pence

K.

3. Re: Is survey design statistically correct?

I don't think the question is actually whether or not your design is correct. It is what it is and it has significant limitations.

I think the question you should be asking is how you are going to be able to use the results you obtain. And given the limitations Karabiner eloquently identified above, I think the answer is that you can't use them to extrapolate beyond the people you survey. You may be able to say that X% of this group felt better after the course, felt less pain, etc. But you can't generalize that to say "people who took a four-day intensive yoga course experience less pain, better sleep, and improved relaxation." It's important to understand and communicate that limitation to your audience.

4. Re: Is survey design statistically correct?

A design is not 'statistically' correct, it is either correct in terms of the methods or its not.

It would be far better to track the same people before and after. If you track a group of people before and a different group after you have no way to know that they are similar on other factors that could cause the result other than your intervention (formally a confound). Which is not really a valid way to compare.

5. Re: Is survey design statistically correct?

Thanks for all replies.

Please remember we're not an academic research organization, nor are the people who join our courses very academic about things. For example, most people who suffer severe lower back pain simply want to get better and feel less such pain, and they are less concerned with whether they improved because of yoga, placebo effect, or some random factor. The important thing for them is that they got better and reduced or got rid of the pain, and they care less about how it happened. It's like a computer, people care less about how it can work and underlying logic and hardware, codes, etc., and more about its actual use. Then, if year after year, the people who participate in our course experience similar positive effects on the tracked variables such as relaxation, sleep quality, etc., they and potential participants in the future can probably draw their own pragmatic conclusion about what causality might be at play.

Now, if we don't want to separate and prove causality by yoga on certain health attributes, but simply want to prove that people who attend our course get better on certain variables (e.g. relaxation, sleep quality, etc.) regardless of what factors might have caused it, would a paired t-test be the correct statistical method to use to give statistical evidence to a statement?

Our sample size is slightly greater than 130 people.

6. Re: Is survey design statistically correct?

Personally, I would not bother with any sort of test. I don't think you will be able to meet the assumptions that are necessary for the test to be valid. Instead, I would limit your statements to descriptive statistics only: In our survey, X% of participants had less pain overall, Y% had less back pain, and Z% reported improved sleep.

Even if you can meet the assumptions required for a statistical test (or you use a non-parametric test), what does it get you? I would say nothing. It doesn't tell you your results are reliable -- you already know they are reliable because you have a limited group. You can't extrapolate your results to the general population even if you do meet the test assumptions, so there's no benefit there. Running more statistics doesn't legitimately tell you anything more than you already know, so I would recommend not bothering. Just stick to the demonstrable facts and leave it at that.

Just my two cents.

7. Re: Is survey design statistically correct?

Originally Posted by Karabiner
You have no control group, therefore you cannot estimate whether pre-post differences are causally related to attending the yoga class.
That depends on the population you are trying to draw inference on. If you are making inference on the population of people who regularly do yoga, then there is no need for a control group.

If you are trying to extend it to the general population (who on average is not familiar with yoga), then yes, you need a control group (taking whatever the normal yoga consists of).

8. Re: Is survey design statistically correct?

Originally Posted by johnsif
Thanks for all replies.

Please remember we're not an academic research organization, nor are the people who join our courses very academic about things. For example, most people who suffer severe lower back pain simply want to get better and feel less such pain, and they are less concerned with whether they improved because of yoga, placebo effect, or some random factor. The important thing for them is that they got better and reduced or got rid of the pain, and they care less about how it happened. It's like a computer, people care less about how it can work and underlying logic and hardware, codes, etc., and more about its actual use. Then, if year after year, the people who participate in our course experience similar positive effects on the tracked variables such as relaxation, sleep quality, etc., they and potential participants in the future can probably draw their own pragmatic conclusion about what causality might be at play.

Now, if we don't want to separate and prove causality by yoga on certain health attributes, but simply want to prove that people who attend our course get better on certain variables (e.g. relaxation, sleep quality, etc.) regardless of what factors might have caused it, would a paired t-test be the correct statistical method to use to give statistical evidence to a statement?

Our sample size is slightly greater than 130 people.
Now, the participant might not care about the why and the how, just the end product, but you the researcher/purveyor most certainly should care.

I don't care how my smart phone works, just that it does. On the other hand, the people who made my smart phone do care.

9. Re: Is survey design statistically correct?

Originally Posted by the42up
That depends on the population you are trying to draw inference on. If you are making inference on the population of people who regularly do yoga, then there is no need for a control group.

If you are trying to extend it to the general population (who on average is not familiar with yoga), then yes, you need a control group (taking whatever the normal yoga consists of).
I was referring to the fact that a simple pe-post comparison does not permit to make causal inferences with regard to some events which took place in between. That has nothing to do with the popultion concerned.

With kind regards

K.

 Tweet

Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts