+ Reply to Thread
Results 1 to 13 of 13

Thread: Power calculation with measurement error on Y

  1. #1
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Power calculation with measurement error on Y




    I'm trying to do a power calculation for a study I'm involved in, and I'm a bit stuck.

    The study is about textual cues of trust. We want to analyze a number of predictors in the text using the LIWC tool (which generates features based on text) and correlate them with perceived trust as determined by the users of this internet platform.

    We want to do a multiple linear regression with a fixed number of predictors (we still have to determine the exact set, but think around 10-15). We expect low to medium correlations (say, .2-.3)

    I want to do a power analysis. It's easy to find material to calculate the power for a correlation (like this)

    However, I'm not sure how to deal with the measurement error. The thing is, the dependent variable is the mean perceived trustworthiness. We estimate this with a sample of ratings for each profile. From a pilot study I know that the SD of these ratings is around 0.6 (out of a Likert-scale of 5). So the SE of the mean rating, and thus the error of my measurement, is estimated to be:

    SE = 0.6/sqrt(a) , where a is the number of raters per profile

    How can I combine the power analysis that uses perfect measurement with the error in measurement to get a definite power?

    And how does the number of predictors come into play?

    So, basically, how do I get from:
    a = number of raters
    n = profiles (sample size)
    p = predictors
    r = expected correlation (0.2)

    to a power? Any help is greatly appreciated!
    Last edited by Junes; 11-01-2016 at 07:30 AM.

  2. #2
    TS Contributor
    Points: 12,287, Level: 72
    Level completed: 60%, Points required for next Level: 163
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,471
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Power calculation with measurement error on Y

    hi,
    off the top of my head, wouldn't the measurement error just act as an increased variance in the data? So, I would expect it to reduce the correlation: if you expect 0.2 based on theory then you might want to plan the sample such that it can detect, say, 0.15 due to measurement errors.
    regards

  3. #3
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: Power calculation with measurement error on Y

    Thanks for your reply. Yeah, that is what I thought too. But how would I calculate the factor? Also, one is in standardized units (r) and the other in unstandardized (SE).

  4. #4
    Devorador de queso
    Points: 95,940, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Power calculation with measurement error on Y

    The mathematician in me says "Yes there is a way to calculate that directly". The lazy person in me says "just simulate it". That's my favorite way to do power analysis.
    I don't have emotions and sometimes that makes me very sad.

  5. The Following 2 Users Say Thank You to Dason For This Useful Post:

    Junes (11-01-2016), rogojel (11-01-2016)

  6. #5
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Power calculation with measurement error on Y

    The assumed correlation coefficient that you use will already represent the "attenuated" correlation due to measurement error, and it is this attenuated correlation coefficient that is directly relevant for power. In other words, most power analyses do not "use perfect measurement," as you say. Knowing the fraction of the observed variance in the DV that is due to measurement error might be interesting, but it does not affect the power analysis in any way.

    As for the influence of the number of other predictors, this has very little influence on power after the partial correlation for the IV of interest (which I assume is the correlation coefficient you're talking about) has been given. All it does is make the denominator degrees of freedom a little smaller. So you can probably safely ignore this altogether, unless your sample size is very small, like less than 20 or something.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  7. The Following User Says Thank You to Jake For This Useful Post:

    Junes (11-01-2016)

  8. #6
    Devorador de queso
    Points: 95,940, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Power calculation with measurement error on Y

    Yeah you could probably safely ignore it. Or you could be awesome and simulate. Either way works.
    I don't have emotions and sometimes that makes me very sad.

  9. #7
    Human
    Points: 12,686, Level: 73
    Level completed: 59%, Points required for next Level: 164
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,363
    Thanks
    455
    Thanked 462 Times in 402 Posts

    Re: Power calculation with measurement error on Y

    I haven't followed the discussion so well, but isn't it so that the correlation among predictors will have an influence om the power. So if there is no correlation (as in a designed experiment) some correlation or very high correlation (i.e. multicolinearity) then the power will be influenced, as increasing multicolinearity will increase standard error.

    Isn't it so that the power from the noncentral t-distribution is influenced by the determinant from the full (X'X)-matrix (and that is influenced by multicolinearity)? So, as I understand it, it is not just about that the critical t-value will increase with decreased degrees of freedom.

    Except for the difficulties in actually computing the the power, there is a difficulty in beforehand to imagine what could the correlations be among x-variables, so that the power computation is realistic.

  10. The Following User Says Thank You to GretaGarbo For This Useful Post:

    Junes (11-01-2016)

  11. #8
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Power calculation with measurement error on Y

    Bottom line as well, will there be measurement error in the actual study you are going to conduct or is it exclusive to the preparatory sample!
    Stop cowardice, ban guns!

  12. #9
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Power calculation with measurement error on Y

    Quote Originally Posted by GretaGarbo View Post
    I haven't followed the discussion so well, but isn't it so that the correlation among predictors will have an influence om the power. So if there is no correlation (as in a designed experiment) some correlation or very high correlation (i.e. multicolinearity) then the power will be influenced, as increasing multicolinearity will increase standard error.

    Isn't it so that the power from the noncentral t-distribution is influenced by the determinant from the full (X'X)-matrix (and that is influenced by multicolinearity)? So, as I understand it, it is not just about that the critical t-value will increase with decreased degrees of freedom.
    Yes, but all of this is already accounted for in the partial correlation coefficient. If we were using the simple correlation coefficient then it would be a different story.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  13. #10
    TS Contributor
    Points: 12,287, Level: 72
    Level completed: 60%, Points required for next Level: 163
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,471
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Power calculation with measurement error on Y

    I guess one could have a situation where one expects a given correlation based on theory not previous experience. In this case one would have to factor the measurement error in.

    regards

  14. #11
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: Power calculation with measurement error on Y

    Thanks a lot for the many replies!

    Quote Originally Posted by Dason
    The lazy person in me says "just simulate it". That's my favorite way to do power analysis.
    Excellent idea! I think I'm going to give that a try.

    Quote Originally Posted by jake
    The assumed correlation coefficient that you use will already represent the "attenuated" correlation due to measurement error, and it is this attenuated correlation coefficient that is directly relevant for power. In other words, most power analyses do not "use perfect measurement," as you say. Knowing the fraction of the observed variance in the DV that is due to measurement error might be interesting, but it does not affect the power analysis in any way.
    Thanks, that is reassuring. However, I'm not sure that's applicable in my case because it's a new instrument. The correlations are just what I expect from some related stuff and my tiny pilot sample. Also, my resources are finite: my participants don't get much in the way of incentive. So there is a limited number of ratings I can have them make. So it either means many profiles with few ratings or few profiles with many ratings, and I want to optimize the power.

    Quote Originally Posted by hlsmith
    Bottom line as well, will there be measurement error in the actual study you are going to conduct or is it exclusive to the preparatory sample!
    I think in the study as well, since I want to look for correlations with mean perceived trustworthiness (in the population) and what I get is an estimate of that, based on a number of ratings.
    Last edited by Junes; 11-01-2016 at 04:31 PM.

  15. #12
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: Power calculation with measurement error on Y

    Another question: as participants will rate only a selection of the profiles (say 20 out of the 150 I will use in the text analysis), these ratings will be correlated. To make matters worse, it seems likely that the software we are using will only allow us to present batches of items. So participants 1-10 might get profiles 1-20, participants 11-20 might get profiles 21-40, etc.

    How big of a problem is this clustering for the estimate? My hunch is that it's not a really a big problem, as we only use the ratings to estimate the mean trustworthiness. We don't use the individual ratings. But maybe I'm wrong about this.

  16. #13
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: Power calculation with measurement error on Y


    Simulation so far show that spending more than 7-10 ratings on a single profile is a waste of resources, and that the resources can better be spent on extra profiles.

    Thanks Dason, simulating was a really good idea. Gives you a lot of insight in the data, too.

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats