Does covariate must be observed variable?

#1
In my study, I found significant interactions between treatment and order of sessions. Can I put the order variable as covariate in order to control this effect?
 
#3
Hi! I randomly assigned participants into two orders: The first group received the intervention in the first session and the placebo in the second session, while the second group received the intervention in the second session and the placebo in the first session. I found significant interaction between treatment and order: the first group scored higher in the intervention, and the second group scored higher in the placebo.
My question is: Can I "control" the effects of order by entering the order variable as a covariate?
 
Last edited:

rogojel

TS Contributor
#5
Hi! I randomly assigned participants into two orders: The first group received the intervention in the first session and the placebo in the second session, while the second group received the intervention in the second session and the placebo in the second session. I found significant interaction between treatment and order: the first group scored higher in the intervention, and the second group scored higher in the placebo.
hi,
doesn't this look like everybody performed better the first time? I don't really see an interaction here.

regards
 
#6
hlsmith- Thank you for your response!

rogojel- that is correct. the interaction is between treatment (intervention, placebo) to order (intervention first (1), placebo first (2))- for scores for order 1, was higher in the intervention, and the scores for order 2, was higher in the placebo. the meaning, as you say, is that order had effect on the performance. I want to "clean" the effect of order to see the effect of the intervention.
 
Last edited:
#8
How about a simpler model where only the order has an effect?
I'm not interested in the order effect- I assumed there will be order effect, that's the reason I counterbalanced and tried to avoid them.
I'm interested in the effect of intervention. When I put the order as a covariate, I find significant effect of the treatment.
 

ondansetron

TS Contributor
#9
I'm not interested in the order effect- I assumed there will be order effect, that's the reason I counterbalanced and tried to avoid them.
I'm interested in the effect of intervention. When I put the order as a covariate, I find significant effect of the treatment.
As you noted before, you should include order as a covariate if you believe it has an actual impact on the dependent variable. You don't need to take much direct interest in it, necessarily. I probably would pay little attention to it if you have a strong theoretical reason to include the order. That is, I wouldn't bother significance testing the order covariate if there is no genuine interest in the order variable aside from trying to estimate other effects. You can avoid any testing errors and the theory/logic should support the order variable's inclusion in the model to properly estimate the other effects. Of course, if there is some interest in the order variable, you can do some testing or interval estimation to learn a bit more about it.

The simplest example I can think of is this: predicting weight in infants as a function of their age in months. A reasonable covariate to include would be the infant's length measurement (similar to accounting for height of someone a bit older). You may not be interested in the effect of length on weight, but to estimate the effect of age on infant weight, length should be accounted for. It is possible an interaction is needed between length and age, but this should be left to your good judgement, theory, and prior literature in most cases.
 
#10
As you noted before, you should include order as a covariate if you believe it has an actual impact on the dependent variable. You don't need to take much direct interest in it, necessarily. I probably would pay little attention to it if you have a strong theoretical reason to include the order. That is, I wouldn't bother significance testing the order covariate if there is no genuine interest in the order variable aside from trying to estimate other effects. You can avoid any testing errors and the theory/logic should support the order variable's inclusion in the model to properly estimate the other effects. Of course, if there is some interest in the order variable, you can do some testing or interval estimation to learn a bit more about it.

The simplest example I can think of is this: predicting weight in infants as a function of their age in months. A reasonable covariate to include would be the infant's length measurement (similar to accounting for height of someone a bit older). You may not be interested in the effect of length on weight, but to estimate the effect of age on infant weight, length should be accounted for. It is possible an interaction is needed between length and age, but this should be left to your good judgement, theory, and prior literature in most cases.
Thanks for your reply! I agree with you. Becasue I randomly assigned the order, the only thing I worried about is the defintion of covariate in some places as observed variable (not manipulated variable).

Also- When I enter the order variable as covariate or as factor, it changes the sum of squares and my F value. Why there are difference? By using covariate, I measure the difference between the levels of my independent variable in the average levels of order. It is not exactly what happens when using the order effect as factor and looking for main effect?
 
Last edited:
#11
Thanks for your reply! I agree with you. Becasue I randomly assigned the order, the only thing I worried about is the defintion of covariate in some places as observed variable (not manipulated variable).

Also- When I enter the order variable as covariate or as factor, it changes the sum of squares and my F value. Why there is difference? By using covariate, I measure the difference between the levels of my independent variable in the average levels of order. It is not exactly what happens when using the order effect as factor and looking for main effect?
Sure, I can see your point. I was just using the term a bit loosely in the sense that you think order impacts the DV but you aren't primarily interested in the order variable. However, due to the effect you believe it has, you're going to include it just to make sure your estimation of other parameters is as optimal as possible for your study. If you know that no matter who is in group 1, they will perform better than group 2, for example, even by randomization, I think you should include the order covariate, although there may be a better approach to this that I haven't yet thought of.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.
 
Last edited:
#12
Sure, I can see your point. I was just using the term a bit loosely in the sense that you think order impacts the DV but you aren't primarily interested in the order variable. However, due to the effect you believe it has, you're going to include it just to make sure your estimation of other parameters is as optimal as possible for your study.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.
Thank you for your helpful answer.
I understand that the main difference is in the intereption of the output, but wonder why the results are different if I put the variable as covariate or as factor, if the mathmatics is the same (the intervention is significant if I put the variable as covariate, but only marginal significant if I put the variable as factor).

Maybe it relates to the manner I entered the order variable? It's a catagorial variable 0f 0,1 - I just entered it as it is.
 
#13
Sure, I can see your point. I was just using the term a bit loosely in the sense that you think order impacts the DV but you aren't primarily interested in the order variable. However, due to the effect you believe it has, you're going to include it just to make sure your estimation of other parameters is as optimal as possible for your study. If you know that no matter who is in group 1, they will perform better than group 2, for example, even by randomization, I think you should include the order covariate, although there may be a better approach to this that I haven't yet thought of.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.
By the way, I didn't find main effect for order, but I find interaction between order and treatment. Entering the order as covariate still deal with it?

the graph of the interaction is attached- the horizantal axis represent the intervention, and the lines represent the order.
 

hlsmith

Omega Contributor
#14
The scale of the y axis will make a visual difference! Can you slap confidence whiskers on the plots and post the model output to clear things up.
 

CowboyBear

Super Moderator
#15
Hi! I randomly assigned participants into two orders: The first group received the intervention in the first session and the placebo in the second session, while the second group received the intervention in the second session and the placebo in the first session. I found significant interaction between treatment and order: the first group scored higher in the intervention, and the second group scored higher in the placebo.
My question is: Can I "control" the effects of order by entering the order variable as a covariate?
Hi all, just a quick response and I haven't carefully read all the replies, but... If order is randomly assigned, you do not need to control for it to accurately estimate the effect of treatment. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would frequently be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.

Note also that if you include main effects or treatment and order plus their interaction, the main effect of treatment now means something different - i.e., it is now the effect of treatment for people in just one of the orders, not both. This is no longer estimating the effect you are interested in.

If the effect of order was of genuine substantive interest in of itself you could include it, but that's clearly not the case here.

When I put the order as a covariate, I find significant effect of the treatment.
Choosing a model specification on the basis that it produces the result you want to see is actually p-hacking and a really bad idea (though I'm sure that wasn't your intention). I strongly suggest that you report the model you originally planned, not this one. In future pre-register your analysis plans before collecting data; that way readers of your research can have faith that the results represent a fair test of the hypothesis of interest.
 
#16
Hi all, just a quick response and I haven't carefully read all the replies, but... If order is randomly assigned, you do not need to control for it to accurately estimate the effect of treatment. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would frequently be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.

Note also that if you include main effects or treatment and order plus their interaction, the main effect of treatment now means something different - i.e., it is now the effect of treatment for people in just one of the orders, not both. This is no longer estimating the effect you are interested in.

If the effect of order was of genuine substantive interest in of itself you could include it, but that's clearly not the case here.



Choosing a model specification on the basis that it produces the result you want to see is actually p-hacking and a really bad idea (though I'm sure that wasn't your intention). I strongly suggest that you report the model you originally planned, not this one. In future pre-register your analysis plans before collecting data; that way readers of your research can have faith that the results represent a fair test of the hypothesis of interest.
Thanks for your reply. The counterbalance deal with the order effect (there was no main effect for order) but still there is a interaction between order and treatment in way that mask the effect of treatment.
You wrote that I need to control "only things that are correlated with the IV and that affect the DV"- as far as I know, covariate cannot be variable that correlate with the IV (this is one of the assumption in ANCOVA)

Alsom you wrote that main effect of treatment now means the effect of treatment for people in just one of the orders, not both- I don't understand this- the effect of treatments means the effect of treatment in the average level of order (I checked it in the descriptive).

Before I analyzed the data, I assume there will be interaction between order and session, and I wanted to "clean" the effect of order by putting it in the between-subject factor or in the covariate factor. Because i'm not interested in the order itself, I decided to put it in the covariate. What do you think?
 

CowboyBear

Super Moderator
#17
Thanks for your reply. The counterbalance deal with the order effect (there was no main effect for order) but still there is a interaction between order and treatment in way that mask the effect of treatment.
If the treatment actually has a main effect, and your design has adequate power, you would still be able to detect this effect without specifying the order*treatment interaction.


You wrote that I need to control "only things that are correlated with the IV and that affect the DV"- as far as I know, covariate cannot be variable that correlate with the IV (this is one of the assumption in ANCOVA)
This is not an assumption of ANCOVA. The assumptions of regression models (including ANCOVA) are covered in this paper: http://pareonline.net/getvn.asp?v=18&n=11

Btw, I'm a little confused as to how you're including an interaction if you're specifying order as a covariate in SPSS(?) I would've thought you needed to include order in the fixed effect field if you wanted to specify the interaction. Are you sure your model includes the interaction term?

Alsom you wrote that main effect of treatment now means the effect of treatment for people in just one of the orders, not both- I don't understand this- the effect of treatments means the effect of treatment in the average level of order (I checked it in the descriptive).
For a default specification, the main effect of treatment (in a model including order and order*treatment) would be the effect of treatment for participants in the first level of order. It is possible to set up your coding such that the main effect is displayed for a participant with an "average level of order", but SPSS wouldn't be doing this by default as far as I know. Even if coded that way, it wouldn't be the same thing as the average effect of the treatment (which is presumably what you are most interested in).

Before I analyzed the data, I assume there will be interaction between order and session, and I wanted to "clean" the effect of order by putting it in the between-subject factor or in the covariate factor. Because i'm not interested in the order itself, I decided to put it in the covariate. What do you think?
You don't need to "clean" the effect of order - your use of randomisation is enough for an unbiased estimate of the effect of treatment. I would suggest that you just estimate the main effect and exclude order (though you can show descriptive statistics displaying the unusual pattern of change).
 
#18
CowboyBear;201749If order is randomly assigned said:
you do not need to control for it to accurately estimate the effect of treatment[/B]. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would frequently be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.
I definitely agree with your post. I also may not have understood what OP meant nor articulated clearly what I meant. I took OP to mean that randomization will not reduce a systematic issue as it normally does.
 

hlsmith

Omega Contributor
#19
Once again can you provide output, also can you label the figure which group is placebo and treatment, no brain is getting stuck.
 
#20
Btw, I'm a little confused as to how you're including an interaction if you're specifying order as a covariate in SPSS(?) I would've thought you needed to include order in the fixed effect field if you wanted to specify the interaction. Are you sure your model includes the interaction term?
I'm not including an interaction. There is significant interaction (if I put the order variable as factor), and that's the reason I want to control it.

This is not an assumption of ANCOVA. The assumptions of regression models (including ANCOVA) are covered in this paper: http://pareonline.net/getvn.asp?v=18&n=11
I know that one of the assumptions of ANCOVA is that the covariate is independent of the treatment effect (the covariant and independent variable are independent). That's the rational behind Lord's paradox.

http://www.real-statistics.com/analysis-of-covariance-ancova/assumptions-ancova/

For a default specification, the main effect of treatment (in a model including order and order*treatment) would be the effect of treatment for participants in the first level of order. It is possible to set up your coding such that the main effect is displayed for a participant with an "average level of order", but SPSS wouldn't be doing this by default as far as I know. Even if coded that way, it wouldn't be the same thing as the average effect of the treatment (which is presumably what you are most interested in).
When I specify the order as covariate, it shows me the main effect of treatment in the average level of order (that's actually what i'm most interested in). Why it isn't the same as the average effect of treatment?


You don't need to "clean" the effect of order - your use of randomisation is enough for an unbiased estimate of the effect of treatment. I would suggest that you just estimate the main effect and exclude order (though you can show descriptive statistics displaying the unusual pattern of change).
I can understand your point, but in this kind of experiment the order has enormous effect on finding. The order effect is twice larger than my intervention effect- that's mean that my intervention have no meaning or effect?

This article claim that we need to control for prognostic covariates regardless of whether they show imbalances: http://egap.org/methods-guides/10-things-know-about-covariate-adjustment
If I have clear rational to include the order as covariate, and to examine differences in the averge level of order, why is that a problem?

Also, I wonder what is the difference between entering order as covariate when the order is randomized, and between entering gender/age as covariate when the gender/age are matched. I know that latter is pretty common, so I wonder what is the difference?

Thanks again
 
Last edited: