+ Reply to Thread
Results 1 to 9 of 9

Thread: Is survey design statistically correct?

  1. #1
    Points: 11, Level: 1
    Level completed: 21%, Points required for next Level: 39

    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Is survey design statistically correct?




    I want to track improvements in how yoga class participants feel on various variables (e.g. energy level, sleep quality, pain felt, mental relaxation) before and after a four-day intensive training course.

    The design is to give each participant a questionnaire at the start of the course where they answer how they feel currently about each tracked variable, for example their energy level, sleep quality, extent of pain, level of mental relaxation, on a scale of 1 to 10.

    Then, after the course, the same participants will answer the same questions again, to evaluate whether they improved on any variable.
    So, the answers are not anonymous, and we can know who answered what at the start as well as the end of the course. We are confident that there will be no problem with the response rate in spite of the answers not being anonymous.

    Is my study design statistically correct? Or how could it be improved?

    And after collecting the data, should I do the paired t-test to check the statistical significance of the differences at the start and after the course for each tracked variable? Is this the right test?

    Thanks for any help!

  2. #2
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Is survey design statistically correct?

    The design is to give each participant a questionnaire at the start of the course where they answer how they feel currently about each tracked variable, for example their energy level, sleep quality, extent of pain, level of mental relaxation, on a scale of 1 to 10.
    Is there any possibility to create a control condition, e.g. a waiting-list control group? Or, something like a "placebo" condition, i.e. a course which somehow resembles the yoga course but hasn't the crucial yoga elements included.
    We are confident that there will be no problem with the response rate in spite of the answers not being anonymous.
    You maybe could ask participants to create a code for themselves (something like "first 2 letters of mother's forename + first 2 letters of father's forename + last 2 digits of postal code of the location where one was born").
    Is my study design statistically correct? Or how could it be improved?
    You have no control group, therefore you cannot estimate whether pre-post differences are causally related to attending the yoga class.

    And after collecting the data, should I do the paired t-test to check the statistical significance of the differences at the start and after the course for each tracked variable? Is this the right test?
    Seems so. But you did not mention how large your sample size is. And, is this just 1 yoga class, or are several classes involved?
    And you'd usually need to survey some additional characteristics, in order to describe the sample
    (age, gender, maybe pre-experience with yoga and the like...).

    Just my 2pence

    K.

  3. #3
    Points: 7,236, Level: 56
    Level completed: 44%, Points required for next Level: 114

    Posts
    299
    Thanks
    9
    Thanked 42 Times in 39 Posts

    Re: Is survey design statistically correct?

    I don't think the question is actually whether or not your design is correct. It is what it is and it has significant limitations.

    I think the question you should be asking is how you are going to be able to use the results you obtain. And given the limitations Karabiner eloquently identified above, I think the answer is that you can't use them to extrapolate beyond the people you survey. You may be able to say that X% of this group felt better after the course, felt less pain, etc. But you can't generalize that to say "people who took a four-day intensive yoga course experience less pain, better sleep, and improved relaxation." It's important to understand and communicate that limitation to your audience.

  4. #4
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Is survey design statistically correct?

    A design is not 'statistically' correct, it is either correct in terms of the methods or its not.

    It would be far better to track the same people before and after. If you track a group of people before and a different group after you have no way to know that they are similar on other factors that could cause the result other than your intervention (formally a confound). Which is not really a valid way to compare.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  5. #5
    Points: 11, Level: 1
    Level completed: 21%, Points required for next Level: 39

    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Is survey design statistically correct?

    Thanks for all replies.

    Please remember we're not an academic research organization, nor are the people who join our courses very academic about things. For example, most people who suffer severe lower back pain simply want to get better and feel less such pain, and they are less concerned with whether they improved because of yoga, placebo effect, or some random factor. The important thing for them is that they got better and reduced or got rid of the pain, and they care less about how it happened. It's like a computer, people care less about how it can work and underlying logic and hardware, codes, etc., and more about its actual use. Then, if year after year, the people who participate in our course experience similar positive effects on the tracked variables such as relaxation, sleep quality, etc., they and potential participants in the future can probably draw their own pragmatic conclusion about what causality might be at play.

    Now, if we don't want to separate and prove causality by yoga on certain health attributes, but simply want to prove that people who attend our course get better on certain variables (e.g. relaxation, sleep quality, etc.) regardless of what factors might have caused it, would a paired t-test be the correct statistical method to use to give statistical evidence to a statement?

    Our sample size is slightly greater than 130 people.

  6. #6
    Points: 7,236, Level: 56
    Level completed: 44%, Points required for next Level: 114

    Posts
    299
    Thanks
    9
    Thanked 42 Times in 39 Posts

    Re: Is survey design statistically correct?

    Personally, I would not bother with any sort of test. I don't think you will be able to meet the assumptions that are necessary for the test to be valid. Instead, I would limit your statements to descriptive statistics only: In our survey, X% of participants had less pain overall, Y% had less back pain, and Z% reported improved sleep.

    Even if you can meet the assumptions required for a statistical test (or you use a non-parametric test), what does it get you? I would say nothing. It doesn't tell you your results are reliable -- you already know they are reliable because you have a limited group. You can't extrapolate your results to the general population even if you do meet the test assumptions, so there's no benefit there. Running more statistics doesn't legitimately tell you anything more than you already know, so I would recommend not bothering. Just stick to the demonstrable facts and leave it at that.

    Just my two cents.

  7. #7
    Points: 3,006, Level: 33
    Level completed: 71%, Points required for next Level: 44

    Posts
    177
    Thanks
    1
    Thanked 29 Times in 29 Posts

    Re: Is survey design statistically correct?

    Quote Originally Posted by Karabiner View Post
    You have no control group, therefore you cannot estimate whether pre-post differences are causally related to attending the yoga class.
    That depends on the population you are trying to draw inference on. If you are making inference on the population of people who regularly do yoga, then there is no need for a control group.

    If you are trying to extend it to the general population (who on average is not familiar with yoga), then yes, you need a control group (taking whatever the normal yoga consists of).

  8. #8
    Points: 3,006, Level: 33
    Level completed: 71%, Points required for next Level: 44

    Posts
    177
    Thanks
    1
    Thanked 29 Times in 29 Posts

    Re: Is survey design statistically correct?

    Quote Originally Posted by johnsif View Post
    Thanks for all replies.

    Please remember we're not an academic research organization, nor are the people who join our courses very academic about things. For example, most people who suffer severe lower back pain simply want to get better and feel less such pain, and they are less concerned with whether they improved because of yoga, placebo effect, or some random factor. The important thing for them is that they got better and reduced or got rid of the pain, and they care less about how it happened. It's like a computer, people care less about how it can work and underlying logic and hardware, codes, etc., and more about its actual use. Then, if year after year, the people who participate in our course experience similar positive effects on the tracked variables such as relaxation, sleep quality, etc., they and potential participants in the future can probably draw their own pragmatic conclusion about what causality might be at play.

    Now, if we don't want to separate and prove causality by yoga on certain health attributes, but simply want to prove that people who attend our course get better on certain variables (e.g. relaxation, sleep quality, etc.) regardless of what factors might have caused it, would a paired t-test be the correct statistical method to use to give statistical evidence to a statement?

    Our sample size is slightly greater than 130 people.
    Now, the participant might not care about the why and the how, just the end product, but you the researcher/purveyor most certainly should care.

    I don't care how my smart phone works, just that it does. On the other hand, the people who made my smart phone do care.

  9. #9
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Is survey design statistically correct?


    Quote Originally Posted by the42up View Post
    That depends on the population you are trying to draw inference on. If you are making inference on the population of people who regularly do yoga, then there is no need for a control group.

    If you are trying to extend it to the general population (who on average is not familiar with yoga), then yes, you need a control group (taking whatever the normal yoga consists of).
    I was referring to the fact that a simple pe-post comparison does not permit to make causal inferences with regard to some events which took place in between. That has nothing to do with the popultion concerned.

    With kind regards

    K.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats