+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 16 to 22 of 22

Thread: Does covariate must be observed variable?

  1. #16
    Points: 311, Level: 6
    Level completed: 22%, Points required for next Level: 39

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Does covariate must be observed variable?




    Quote Originally Posted by CowboyBear View Post
    Hi all, just a quick response and I haven't carefully read all the replies, but... If order is randomly assigned, you do not need to control for it to accurately estimate the effect of treatment. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would frequently be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.

    Note also that if you include main effects or treatment and order plus their interaction, the main effect of treatment now means something different - i.e., it is now the effect of treatment for people in just one of the orders, not both. This is no longer estimating the effect you are interested in.

    If the effect of order was of genuine substantive interest in of itself you could include it, but that's clearly not the case here.



    Choosing a model specification on the basis that it produces the result you want to see is actually p-hacking and a really bad idea (though I'm sure that wasn't your intention). I strongly suggest that you report the model you originally planned, not this one. In future pre-register your analysis plans before collecting data; that way readers of your research can have faith that the results represent a fair test of the hypothesis of interest.
    Thanks for your reply. The counterbalance deal with the order effect (there was no main effect for order) but still there is a interaction between order and treatment in way that mask the effect of treatment.
    You wrote that I need to control "only things that are correlated with the IV and that affect the DV"- as far as I know, covariate cannot be variable that correlate with the IV (this is one of the assumption in ANCOVA)

    Alsom you wrote that main effect of treatment now means the effect of treatment for people in just one of the orders, not both- I don't understand this- the effect of treatments means the effect of treatment in the average level of order (I checked it in the descriptive).

    Before I analyzed the data, I assume there will be interaction between order and session, and I wanted to "clean" the effect of order by putting it in the between-subject factor or in the covariate factor. Because i'm not interested in the order itself, I decided to put it in the covariate. What do you think?

  2. #17
    TS Contributor
    Points: 19,199, Level: 87
    Level completed: 70%, Points required for next Level: 151
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,080
    Thanks
    123
    Thanked 429 Times in 330 Posts

    Re: Does covariate must be observed variable?

    Quote Originally Posted by PsychologyS View Post
    Thanks for your reply. The counterbalance deal with the order effect (there was no main effect for order) but still there is a interaction between order and treatment in way that mask the effect of treatment.
    If the treatment actually has a main effect, and your design has adequate power, you would still be able to detect this effect without specifying the order*treatment interaction.


    You wrote that I need to control "only things that are correlated with the IV and that affect the DV"- as far as I know, covariate cannot be variable that correlate with the IV (this is one of the assumption in ANCOVA)
    This is not an assumption of ANCOVA. The assumptions of regression models (including ANCOVA) are covered in this paper: http://pareonline.net/getvn.asp?v=18&n=11

    Btw, I'm a little confused as to how you're including an interaction if you're specifying order as a covariate in SPSS(?) I would've thought you needed to include order in the fixed effect field if you wanted to specify the interaction. Are you sure your model includes the interaction term?

    Alsom you wrote that main effect of treatment now means the effect of treatment for people in just one of the orders, not both- I don't understand this- the effect of treatments means the effect of treatment in the average level of order (I checked it in the descriptive).
    For a default specification, the main effect of treatment (in a model including order and order*treatment) would be the effect of treatment for participants in the first level of order. It is possible to set up your coding such that the main effect is displayed for a participant with an "average level of order", but SPSS wouldn't be doing this by default as far as I know. Even if coded that way, it wouldn't be the same thing as the average effect of the treatment (which is presumably what you are most interested in).

    Before I analyzed the data, I assume there will be interaction between order and session, and I wanted to "clean" the effect of order by putting it in the between-subject factor or in the covariate factor. Because i'm not interested in the order itself, I decided to put it in the covariate. What do you think?
    You don't need to "clean" the effect of order - your use of randomisation is enough for an unbiased estimate of the effect of treatment. I would suggest that you just estimate the main effect and exclude order (though you can show descriptive statistics displaying the unusual pattern of change).
    Matt aka CB | twitter.com/matthewmatix

  3. #18
    TS Contributor
    Points: 1,905, Level: 26
    Level completed: 5%, Points required for next Level: 95

    Posts
    253
    Thanks
    39
    Thanked 71 Times in 62 Posts

    Re: Does covariate must be observed variable?

    Quote Originally Posted by CowboyBear;201749If order is randomly assigned, [B
    you do not need to control for it to accurately estimate the effect of treatment[/B]. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would frequently be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.
    I definitely agree with your post. I also may not have understood what OP meant nor articulated clearly what I meant. I took OP to mean that randomization will not reduce a systematic issue as it normally does.

  4. #19
    Omega Contributor
    Points: 39,072, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,075
    Thanks
    402
    Thanked 1,192 Times in 1,153 Posts

    Re: Does covariate must be observed variable?

    Once again can you provide output, also can you label the figure which group is placebo and treatment, no brain is getting stuck.
    Stop cowardice, ban guns!

  5. #20
    Points: 311, Level: 6
    Level completed: 22%, Points required for next Level: 39

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Does covariate must be observed variable?

    Quote Originally Posted by CowboyBear View Post
    Btw, I'm a little confused as to how you're including an interaction if you're specifying order as a covariate in SPSS(?) I would've thought you needed to include order in the fixed effect field if you wanted to specify the interaction. Are you sure your model includes the interaction term?
    I'm not including an interaction. There is significant interaction (if I put the order variable as factor), and that's the reason I want to control it.

    This is not an assumption of ANCOVA. The assumptions of regression models (including ANCOVA) are covered in this paper: http://pareonline.net/getvn.asp?v=18&n=11
    I know that one of the assumptions of ANCOVA is that the covariate is independent of the treatment effect (the covariant and independent variable are independent). That's the rational behind Lord's paradox.

    http://www.real-statistics.com/analy...ptions-ancova/

    Quote Originally Posted by CowboyBear View Post
    For a default specification, the main effect of treatment (in a model including order and order*treatment) would be the effect of treatment for participants in the first level of order. It is possible to set up your coding such that the main effect is displayed for a participant with an "average level of order", but SPSS wouldn't be doing this by default as far as I know. Even if coded that way, it wouldn't be the same thing as the average effect of the treatment (which is presumably what you are most interested in).
    When I specify the order as covariate, it shows me the main effect of treatment in the average level of order (that's actually what i'm most interested in). Why it isn't the same as the average effect of treatment?


    Quote Originally Posted by CowboyBear View Post
    You don't need to "clean" the effect of order - your use of randomisation is enough for an unbiased estimate of the effect of treatment. I would suggest that you just estimate the main effect and exclude order (though you can show descriptive statistics displaying the unusual pattern of change).
    I can understand your point, but in this kind of experiment the order has enormous effect on finding. The order effect is twice larger than my intervention effect- that's mean that my intervention have no meaning or effect?

    This article claim that we need to control for prognostic covariates regardless of whether they show imbalances: http://egap.org/methods-guides/10-th...ate-adjustment
    If I have clear rational to include the order as covariate, and to examine differences in the averge level of order, why is that a problem?

    Also, I wonder what is the difference between entering order as covariate when the order is randomized, and between entering gender/age as covariate when the gender/age are matched. I know that latter is pretty common, so I wonder what is the difference?

    Thanks again
    Last edited by PsychologyS; 07-05-2017 at 08:31 AM.

  6. #21
    Points: 311, Level: 6
    Level completed: 22%, Points required for next Level: 39

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Does covariate must be observed variable?

    Quote Originally Posted by hlsmith View Post
    Once again can you provide output, also can you label the figure which group is placebo and treatment, no brain is getting stuck.
    The graph with the labels attached.
    Attached Images  

  7. #22
    TS Contributor
    Points: 19,199, Level: 87
    Level completed: 70%, Points required for next Level: 151
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,080
    Thanks
    123
    Thanked 429 Times in 330 Posts

    Re: Does covariate must be observed variable?


    Quote Originally Posted by PsychologyS View Post
    I'm not including an interaction. There is significant interaction (if I put the order variable as factor), and that's the reason I want to control it.
    If you include the order variable as a covariate (and don't specify an interaction), then you actually aren't controlling for the interaction - you're controlling for a difference in average scores between the two orders.

    I know that one of the assumptions of ANCOVA is that the covariate is independent of the treatment effect (the covariant and independent variable are independent). That's the rational behind Lord's paradox.

    http://www.real-statistics.com/analy...ptions-ancova/
    This is interesting stuff. I think you're referring to this paper; it is true that controlling for a covariate is a bad way to get a causal estimate of the effect of the treatment if the treatment affects the covariate, but this is actually just a special case of an issue that's not specific to ANCOVA: That you shouldn't control for a variable that's affected by the treatment. (So the issue isn't dependence per se but the direction of the causal relationship). The article sort-of addresses this on page 45. (Though this isn't probably a big concern relative to your question here).

    This article claim that we need to control for prognostic covariates regardless of whether they show imbalances: http://egap.org/methods-guides/10-th...ate-adjustment
    There's a lot of disagreement about this in the literature... note the bit on that page asking "Assuming that random assignment was implemented correctly, should examination of imbalances play any role in choosing which covariates to adjust for?"

    If I have clear rational to include the order as covariate, and to examine differences in the averge level of order, why is that a problem?
    The worry I have is that your desire to specify the order variable as a covariate is probably somewhat influenced by your knowledge that doing so produces a significant effect of treatment. Imagine: If the treatment had shown a significant effect when not controlling for order, but no significant effect when you did control for order, what would you be reporting? I'm pretty sure you'd be saying "participants are randomly assigned to orders - I don't need to control for order", and that would sound reasonable too, wouldn't it? I'm not saying you're trying to do anything misleading, but it's very hard to avoid being influenced by knowledge of the results, no matter how hard we try!

    This problem is the garden of forking paths, and it leads to bias in the reporting of results. Whether a test produces the result we want to see should never have any influence on whether we choose to report that test, and the best way to avoid this is by publicly committing to an analysis strategy before looking at the data, via a pre-registration.

    If you think controlling for order is the right thing to do - despite the randomisation by design avoiding any systematic effect of order on average scores - then you can pre-register a replication of your study specifying that you will do so, and run the replication.

    In the absence of a pre-registration, you might be best off reporting your analysis in several different ways, and being very clear that treatment doesn't have a significant effect for some reasonable choices of analysis (and as such there isn't strong evidence that the treatment works).
    Matt aka CB | twitter.com/matthewmatix

  8. The Following User Says Thank You to CowboyBear For This Useful Post:

    ondansetron (07-06-2017)

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats