- Thread starter PsychologyS
- Start date

Hi! I randomly assigned participants into two orders: The first group received the intervention in the first session and the placebo in the second session, while the second group received the intervention in the second session and the placebo in the first session. I found significant interaction between treatment and order: the first group scored higher in the intervention, and the second group scored higher in the placebo.

My question is: Can I "control" the effects of order by entering the order variable as a covariate?

My question is: Can I "control" the effects of order by entering the order variable as a covariate?

Last edited:

Hi! I randomly assigned participants into two orders: The first group received the intervention in the first session and the placebo in the second session, while the second group received the intervention in the second session and the placebo in the second session. I found significant interaction between treatment and order: the first group scored higher in the intervention, and the second group scored higher in the placebo.

doesn't this look like everybody performed better the first time? I don't really see an interaction here.

regards

hlsmith- Thank you for your response!

rogojel- that is correct. the interaction is between treatment (intervention, placebo) to order (intervention first (1), placebo first (2))- for scores for order 1, was higher in the intervention, and the scores for order 2, was higher in the placebo. the meaning, as you say, is that order had effect on the performance. I want to "clean" the effect of order to see the effect of the intervention.

rogojel- that is correct. the interaction is between treatment (intervention, placebo) to order (intervention first (1), placebo first (2))- for scores for order 1, was higher in the intervention, and the scores for order 2, was higher in the placebo. the meaning, as you say, is that order had effect on the performance. I want to "clean" the effect of order to see the effect of the intervention.

Last edited:

How about a simpler model where only the order has an effect?

I'm interested in the effect of intervention. When I put the order as a covariate, I find significant effect of the treatment.

I'm not interested in the order effect- I assumed there will be order effect, that's the reason I counterbalanced and tried to avoid them.

I'm interested in the effect of intervention. When I put the order as a covariate, I find significant effect of the treatment.

I'm interested in the effect of intervention. When I put the order as a covariate, I find significant effect of the treatment.

The simplest example I can think of is this: predicting weight in infants as a function of their age in months. A reasonable covariate to include would be the infant's length measurement (similar to accounting for height of someone a bit older). You may not be interested in the effect of length on weight, but to estimate the effect of age on infant weight, length should be accounted for. It is possible an interaction is needed between length and age, but this should be left to your good judgement, theory, and prior literature in most cases.

As you noted before, you should include order as a covariate if you believe it has an actual impact on the dependent variable. You don't need to take much direct interest in it, necessarily. I probably would pay little attention to it if you have a strong theoretical reason to include the order. That is, I wouldn't bother significance testing the order covariate if there is no genuine interest in the order variable aside from trying to estimate other effects. You can avoid any testing errors and the theory/logic should support the order variable's inclusion in the model to properly estimate the other effects. Of course, if there is some interest in the order variable, you can do some testing or interval estimation to learn a bit more about it.

The simplest example I can think of is this: predicting weight in infants as a function of their age in months. A reasonable covariate to include would be the infant's length measurement (similar to accounting for height of someone a bit older). You may not be interested in the effect of length on weight, but to estimate the effect of age on infant weight, length should be accounted for. It is possible an interaction is needed between length and age, but this should be left to your good judgement, theory, and prior literature in most cases.

The simplest example I can think of is this: predicting weight in infants as a function of their age in months. A reasonable covariate to include would be the infant's length measurement (similar to accounting for height of someone a bit older). You may not be interested in the effect of length on weight, but to estimate the effect of age on infant weight, length should be accounted for. It is possible an interaction is needed between length and age, but this should be left to your good judgement, theory, and prior literature in most cases.

Also- When I enter the order variable as covariate or as factor, it changes the sum of squares and my F value. Why there are difference? By using covariate, I measure the difference between the levels of my independent variable in the average levels of order. It is not exactly what happens when using the order effect as factor and looking for main effect?

Last edited:

Thanks for your reply! I agree with you. Becasue I randomly assigned the order, the only thing I worried about is the defintion of covariate in some places as observed variable (not manipulated variable).

Also- When I enter the order variable as covariate or as factor, it changes the sum of squares and my F value. Why there is difference? By using covariate, I measure the difference between the levels of my independent variable in the average levels of order. It is not exactly what happens when using the order effect as factor and looking for main effect?

Also- When I enter the order variable as covariate or as factor, it changes the sum of squares and my F value. Why there is difference? By using covariate, I measure the difference between the levels of my independent variable in the average levels of order. It is not exactly what happens when using the order effect as factor and looking for main effect?

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.

Last edited:

Sure, I can see your point. I was just using the term a bit loosely in the sense that you think order impacts the DV but you aren't primarily interested in the order variable. However, due to the effect you believe it has, you're going to include it just to make sure your estimation of other parameters is as optimal as possible for your study.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.

I understand that the main difference is in the intereption of the output, but wonder why the results are different if I put the variable as covariate or as factor, if the mathmatics is the same (the intervention is significant if I put the variable as covariate, but only marginal significant if I put the variable as factor).

Maybe it relates to the manner I entered the order variable? It's a catagorial variable 0f 0,1 - I just entered it as it is.

Sure, I can see your point. I was just using the term a bit loosely in the sense that you think order impacts the DV but you aren't primarily interested in the order variable. However, due to the effect you believe it has, you're going to include it just to make sure your estimation of other parameters is as optimal as possible for your study. If you know that no matter who is in group 1, they will perform better than group 2, for example, even by randomization, I think you should include the order covariate, although there may be a better approach to this that I haven't yet thought of.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.

For your second question, I believe the only real difference is how you, the researcher, will choose to interpret the output. The mathematics is the same in this case, but you are aware you're treating the order as a covariate rather than a variable of interest. I think some software packages separate out "independent variables/factors" from "covariates" to help researchers keep things separate in their mind.

I fairly certain of this, but maybe someone else can clarify if I misunderstood something.

the graph of the interaction is attached- the horizantal axis represent the intervention, and the lines represent the order.

My question is: Can I "control" the effects of order by entering the order variable as a covariate?

Note also that if you include main effects or treatment and order plus their interaction,

If the effect of order was of genuine substantive interest in of itself you could include it, but that's clearly not the case here.

When I put the order as a covariate, I find significant effect of the treatment.

Hi all, just a quick response and I haven't carefully read all the replies, but... If order is randomly assigned, **you do not need to control for it to accurately estimate the effect of treatment**. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would *frequently* be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.

Note also that if you include main effects or treatment and order plus their interaction,*the main effect of treatment now means something different* - i.e., it is now the effect of treatment for people in just one of the orders, not both. This is no longer estimating the effect you are interested in.

If the effect of order was of genuine substantive interest in of itself you could include it, but that's clearly not the case here.

Choosing a model specification on the basis that it produces the result you want to see is actually p-hacking and a really bad idea (though I'm sure that wasn't your intention). I strongly suggest that you report the model you originally planned, not this one. In future pre-register your analysis plans before collecting data; that way readers of your research can have faith that the results represent a fair test of the hypothesis of interest.

Note also that if you include main effects or treatment and order plus their interaction,

If the effect of order was of genuine substantive interest in of itself you could include it, but that's clearly not the case here.

Choosing a model specification on the basis that it produces the result you want to see is actually p-hacking and a really bad idea (though I'm sure that wasn't your intention). I strongly suggest that you report the model you originally planned, not this one. In future pre-register your analysis plans before collecting data; that way readers of your research can have faith that the results represent a fair test of the hypothesis of interest.

You wrote that I need to control "only things that are correlated with the IV and that affect the DV"- as far as I know, covariate cannot be variable that correlate with the IV (this is one of the assumption in ANCOVA)

Alsom you wrote that main effect of treatment now means the effect of treatment for people in just one of the orders, not both- I don't understand this- the effect of treatments means the effect of treatment in the average level of order (I checked it in the descriptive).

Before I analyzed the data, I assume there will be interaction between order and session, and I wanted to "clean" the effect of order by putting it in the between-subject factor or in the covariate factor. Because i'm not interested in the order itself, I decided to put it in the covariate. What do you think?

Thanks for your reply. The counterbalance deal with the order effect (there was no main effect for order) but still there is a interaction between order and treatment in way that mask the effect of treatment.

You wrote that I need to control "only things that are correlated with the IV and that affect the DV"- as far as I know, covariate cannot be variable that correlate with the IV (this is one of the assumption in ANCOVA)

Btw, I'm a little confused as to how you're including an interaction if you're specifying order as a covariate in SPSS(?) I would've thought you needed to include order in the fixed effect field if you wanted to specify the interaction. Are you sure your model includes the interaction term?

Alsom you wrote that main effect of treatment now means the effect of treatment for people in just one of the orders, not both- I don't understand this- the effect of treatments means the effect of treatment in the average level of order (I checked it in the descriptive).

Before I analyzed the data, I assume there will be interaction between order and session, and I wanted to "clean" the effect of order by putting it in the between-subject factor or in the covariate factor. Because i'm not interested in the order itself, I decided to put it in the covariate. What do you think?

CowboyBear;201749If order is randomly assigned said:

you do not need to control for it to accurately estimate the effect of treatment[/B]. (You don't need to control for everything that affects the DV, only things that are correlated with the IV and that affect the DV - i.e. confounding variables). It would *frequently* be the case that order affects the DV, but the entire point of counterbalancing with random assignment to orders is so that we don't have to worry about order effects confounding the results.

Btw, I'm a little confused as to how you're including an interaction if you're specifying order as a covariate in SPSS(?) I would've thought you needed to include order in the fixed effect field if you wanted to specify the interaction. Are you sure your model includes the interaction term?

This is not an assumption of ANCOVA. The assumptions of regression models (including ANCOVA) are covered in this paper: http://pareonline.net/getvn.asp?v=18&n=11

http://www.real-statistics.com/analysis-of-covariance-ancova/assumptions-ancova/

For a default specification, the main effect of treatment (in a model including order and order*treatment) would be the effect of treatment for participants in the first level of order. It is possible to set up your coding such that the main effect is displayed for a participant with an "average level of order", but SPSS wouldn't be doing this by default as far as I know. Even if coded that way, it wouldn't be the same thing as the average effect of the treatment (which is presumably what you are most interested in).

You don't need to "clean" the effect of order - your use of randomisation is enough for an unbiased estimate of the effect of treatment. I would suggest that you just estimate the main effect and exclude order (though you can show descriptive statistics displaying the unusual pattern of change).

This article claim that we need to control for prognostic covariates regardless of whether they show imbalances: http://egap.org/methods-guides/10-things-know-about-covariate-adjustment

If I have clear rational to include the order as covariate, and to examine differences in the averge level of order, why is that a problem?

Also, I wonder what is the difference between entering order as covariate when the order is randomized, and between entering gender/age as covariate when the gender/age are matched. I know that latter is pretty common, so I wonder what is the difference?

Thanks again

Last edited: