+ Reply to Thread
Results 1 to 4 of 4

Thread: Why "controlling for X" studies may be fatally flawed

  1. #1
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Why "controlling for X" studies may be fatally flawed




    I came across this paper this week, which was quite an eye-opener for me, so I wanted to share.

    We are probably all familiar with "controlling for X" studies where you control for confounds by measuring them and adding them to your model. For instance, today I read an article on a news site about that claimed fecundity suffered from sleeping in, controlling for age, smoking, unemployment and depression. This is an incredibly common design in social science and medical studies.

    The problem however is that where the controlled variable is a latent variable (such as depression), it will not be completely reliably measured. This affects the extent to which it can actually function as a control, and a spurious correlation may be found after all.

    The authors give an example where the number of people drowning correlates with ice cream sales. A common example to show what a confound is: the effect disappears when we add temperature to the model. However, if we would hypothetically ask people "how hot the day is" and this measure has a reliability of 0.4, there is still a substantial correlation between ice cream and drownings "controlling" for hot days.

    The authors show that even with what are considered to be reliable instruments in psychology (e.g., alpha = 0.8), Type I error may double. It even gets worse with large sample sizes. The authors suggest using SEM to model latent variables.

    This potentially puts into question a lot of research, especially in the medical and psychological literature. I never thought about it, but it makes perfect sense.
    Last edited by Junes; 10-19-2016 at 06:44 PM.

  2. #2
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Why "controlling for X" studies may be fatally flawed

    I'm glad you liked it
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  3. The Following User Says Thank You to Jake For This Useful Post:

    Junes (10-19-2016)

  4. #3
    Devorador de queso
    Points: 95,754, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,932
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Why "controlling for X" studies may be fatally flawed

    I'd read it but I don't trust the authors at all (well at least one in particular)
    I don't have emotions and sometimes that makes me very sad.

  5. #4
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: Why "controlling for X" studies may be fatally flawed


    Haha, I didn't expect a reply from one of the authors here :-). But yeah, it's really interesting stuff. I especially liked how the article started with an intuitive example.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats