Queries regarding ANCOVA and MANCOVA

#1
In a pre-post test experimental study having 3x2 factorial design, there are:

1. two independent variables:
(a) teaching methods (3 levels/groups) (b) gender

2. three dependent variables:
(a) achievement (3 sublevels-knowledge, comprehension, application)
(b) attitude (measured by likert scale having 5 subscales)
(c) ) attitude (measured by likert scale having 5 subscales)

Query 1: If the 3 groups are found to be equivalent at pre-test stage (either quasi or true experimental method is used), is it ok to go for ANCOVA using pre-test measures as covariates or is there a need to include intelligence or other relevant variables as covariates?

Query 2: Is it ok if MANCOVA is conducted on each dependent variable with pre-test measures as covariates to detect any significant differences among the 3 groups?(MANCOVA because sublevels/subscales of each dependent variable are correlated with each other)

Query 3: Is it ok if ANCOVA is conducted on each sublevel of each dependent variable with pre-test measures as covariates, by adopting Bonferroni method for controlling the type I error rates?

Query 4: What post hoc tests (in SPSS) can be used in this case?
 
#4
You mean you taught some students with three different methods, and now you want to see how their attitude and achievement level are affected by the methods of teaching and the participants' gender?

ANCOVA and MANCOVA can be useful, but I don't know can you input the pre-test measurements as covariates.

About the need to include the intelligence into your design, of course it would render your results much clearer, but if it is impossible to collect their IQ at this moment, you can still run your tests with current dataset, and discuss the absence of IQ as an important issue in your limitations. Remember that there are always some hidden variables, and it is somehow impossible to include all the possibly affecting variables (such as eyesight, level of motivation, sleep deprivation, and many more) into an experiment on education. So if you even control the IQ, you still have to control many other ones.

About Q2, I think it might be alright, but not because of the correlation between the scales, but because of having more than one dependent variable.

Query 3: Is it ok if ANCOVA is conducted on each sublevel of each dependent variable with pre-test measures as covariates, by adopting Bonferroni method for controlling the type I error rates?
I don't know about the possiblity of including those pre-test data as covariates. But Bonferroni is used for pairwise comparisons, not the ANOVA itself. I think you as well meant this. If so, you can run a Bonferroni posthoc test which automatically controls for type I error.

Q4: Simply open the corresponding dialog box for the test and hit the "post hoc" button. Many available tests are shown, which you can select from them. LSD, Bonferroni, and Tukey are usually used.
 

Dragan

Super Moderator
#5
Q4: Simply open the corresponding dialog box for the test and hit the "post hoc" button. Many available tests are shown, which you can select from them. LSD, Bonferroni, and Tukey are usually used.
@victorxstc: John Tukey's approach is not appropriate in this context because the post-hoc statistic doesn't take into account the differernces between the means and variances of the covariate.
 

CB

Super Moderator
#6
I understand you're really keen to get some help, but I'm not a huge fan of the approach of posters approaching regulars individually with visitor messages to ask for their help. I hadn't made any contribution to this thread, so was a bit mystified by your visitor message. The best way to get responses is to follow the posting guidelines: http://www.talkstats.com/showthread.php/14960-Forum-Guidelines-Smart-posting-behavior-pays-off

Query 1: If the 3 groups are found to be equivalent at pre-test stage (either quasi or true experimental method is used), is it ok to go for ANCOVA using pre-test measures as covariates or is there a need to include intelligence or other relevant variables as covariates?
This is a design question more than a statistical one. Did you use an experimental strategy such as random assignment to render you rexperimental groups equal on extraneous potential confounds like IQ? If not, statistically controlling for IQ might be appropriate (but there could be hundreds of other potential confounds too).

Query 2: Is it ok if MANCOVA is conducted on each dependent variable with pre-test measures as covariates to detect any significant differences among the 3 groups?(MANCOVA because sublevels/subscales of each dependent variable are correlated with each other)
I don't quite follow this. MANCOVA involves multiple dependent variables, it sounds here like you're thinking of analysing the effects on each DV one by one? Or am I misreading?

Query 3: Is it ok if ANCOVA is conducted on each sublevel of each dependent variable with pre-test measures as covariates, by adopting Bonferroni method for controlling the type I error rates?
I don't really follow here. How can you do ANCOVA within a single "sublevel"?
 
#7
@victorxstc: John Tukey's approach is not appropriate in this context because the post-hoc statistic doesn't take into account the differernces between the means and variances of the covariate.
Thanks Dragan for the hint, although I had difficulty in getting to the point as we did not know the sample properties yet. I prefer Bonferroni, but in a recent paper, the reviewer asked me to run a Tukey for a 2-way ANOVA, instead of the Bonferroni. Apparently, it might be popular, despite the shortcomings you mentioned.
 
#8
You mean you taught some students with three different methods, and now you want to see how their attitude and achievement level are affected by the methods of teaching and the participants' gender?

ANCOVA and MANCOVA can be useful, but I don't know can you input the pre-test measurements as covariates.

About the need to include the intelligence into your design, of course it would render your results much clearer, but if it is impossible to collect their IQ at this moment, you can still run your tests with current dataset, and discuss the absence of IQ as an important issue in your limitations. Remember that there are always some hidden variables, and it is somehow impossible to include all the possibly affecting variables (such as eyesight, level of motivation, sleep deprivation, and many more) into an experiment on education. So if you even control the IQ, you still have to control many other ones.

About Q2, I think it might be alright, but not because of the correlation between the scales, but because of having more than one dependent variable.



I don't know about the possiblity of including those pre-test data as covariates. But Bonferroni is used for pairwise comparisons, not the ANOVA itself. I think you as well meant this. If so, you can run a Bonferroni posthoc test which automatically controls for type I error.

Q4: Simply open the corresponding dialog box for the test and hit the "post hoc" button. Many available tests are shown, which you can select from them. LSD, Bonferroni, and Tukey are usually used.
Thanks a lot for your reply.

1. I am clear about using pre-test measures as covariates (I found this in a number of peer-reviewed journals and books too). I also noticed in a number of research studies that additional covariates are being used, besides pre-test measures as covariates. Is there any maximum limit for the number of covariates to be included?

2. I have 3 dependent variables: Achievement, Attitude and Attitude.
I would like to go for MANCOVA on each of the dependent variable separately because of the following reasons:

a. Achievement is a big construct as it consists of 3 subdimensions/subscales (namely, knowledge, comprehension, application). These subscales are intercorrelated with each other. That’s why, I would like to do MANCOVA on achievement.

b. Besides MANCOVA, I want to compare groups on each sublevel of achievement separately by making use of ANCOVA. Since I will be using ANCOVA 3 times (as there are 3 sublevels), I would like to control type I error rates. To control the possibly inflated Type I error rates, can the Bonferroni method be adopted to limit the familywise Type I error rate to a 0.05 alpha level. Hence, only results significant at the 0.017 level can be accepted (obtained by dividing 0.05 by the number of subscales).

c. Similarly, attitude is a big construct, consisting of 5 subscales which are intercorrelated with each other. That’s why, I would like to do MANCOVA on attitude and ANCOVA on each sublevel of attitude ( in the same way as I mentioned in b).

d. I am still confused about post hoc analysis. Please clarify more.
 
#9
Thanks Dragan for the hint, although I had difficulty in getting to the point as we did not know the sample properties yet. I prefer Bonferroni, but in a recent paper, the reviewer asked me to run a Tukey for a 2-way ANOVA, instead of the Bonferroni. Apparently, it might be popular, despite the shortcomings you mentioned.
I did not get what you exactly mean.
 
#10
1. I am clear about using pre-test measures as covariates (I found this in a number of peer-reviewed journals and books too). I also noticed in a number of research studies that additional covariates are being used, besides pre-test measures as covariates. Is there any maximum limit for the number of covariates to be included?
Ok. glad to hear that. But I don't know the maximum. Somewhere else I remember some experts (I think it was Dason) said that a regression can theoretically accept n-1 covariables, where n = sample size. So since ANCOVA is a regression analysis, such number can be applied to it too.

2. I have 3 dependent variables: Achievement, Attitude and Attitude.
I would like to go for MANCOVA on each of the dependent variable separately because of the following reasons:

a. Achievement is a big construct as it consists of 3 subdimensions/subscales (namely, knowledge, comprehension, application). These subscales are intercorrelated with each other. That’s why, I would like to do MANCOVA on achievement.
I think that's fine to use a MANCOVA here, because of the number of your dependent variables. Please note that here, your actual variables are those subscales (while the achievement as a total or combined score can be a single variable). But since MANCOVA deals with situations with more than one dependent variable, it is justified in my opinion to use it here.

b. Besides MANCOVA, I want to compare groups on each sublevel of achievement separately by making use of ANCOVA. Since I will be using ANCOVA 3 times (as there are 3 sublevels), I would like to control type I error rates. To control the possibly inflated Type I error rates, can the Bonferroni method be adopted to limit the familywise Type I error rate to a 0.05 alpha level. Hence, only results significant at the 0.017 level can be accepted (obtained by dividing 0.05 by the number of subscales).
Aha :) It is so controversial if the number of tests in a study dictates needing the correction for multiple comparisons, or the number of tests PLUS the type of tests.

I have had the same concern before, and after conversations with guys here (especially I'm so thankful to CowboyBear), and checking many articles with such designs, I can tell you quite confidently that your case (three different ANCOVAs) does not need any correction of the multiple comparison problem. So an alpha set at 0.05 is quite correct in that case.

If you are interested in the reason, actually nobody's sure!! The scientific community has came to accept that "we correct familywise error when doing pair-wise comparisons within an ANOVA framework OR when the number of tests are too much." Is correcting the multiple comparison problem in every situation with more than one test needed? Nobody don't know for sure! But they don't do it in cases other than the one within an ANOVA design, or cases with thousands or millions of parallel tests (such as assessment of MRI images, each of which can have millions of voxels [in the lattter, a Bonferroni is not so practical though, but that is something else]).

If you wanted to consider the number of tests when correcting the multiple comparison problem, you could say "Why only 3 different ANCOVAs? I have some more tests to run in this setup which according to the Bonferroni method, will increase the chance of obtaining a random P value smaller than 0.05. So I have to divide 0.05 not by 3, but by 6 or 7 (or whatever the number of your tests indicates)."... So I recommend you to adhere to the rule of thumb and do correct the multiple comparison thing within each ANOVA, and gladly the post hoc tests already take it into account while giving you P values.

c. Similarly, attitude is a big construct, consisting of 5 subscales which are intercorrelated with each other. That’s why, I would like to do MANCOVA on attitude and ANCOVA on each sublevel of attitude ( in the same way as I mentioned in b).
Ok. I think you can do both of them. Either consider each of those subscales as actual variables and inputing them into a MANCOVA (because the dependents are > 1, not because they are correlated. If they were not correlated, you still had to do a MANCOVA). Or, combine them all and calculate an "attitude" score and evaluate it using a single ANCOVA. Again you don't need and should not correct for multiple comparison thing here.

d. I am still confused about post hoc analysis. Please clarify more.
I don't know what aspect of it still confuses you. I think you know what it is, but if not, I can tell you that an ANOVA (or its siblings) tell us whether there is any significant difference in the whole setup or not. Once an ANOVA returned a significant P value, we are sure that there is some difference among the variables involved in that ANOVA. But we want to be more specific, and we want to understand which variables are the ones responsible for the significant ANOVA P value? So we run pairwise tests between different levels of different variables to see which of these correspond to the total ANOVA significance. There are some specific tests designed for this purpose. They can both compare two groups and correct the multiple comparison problem inherently (so using them spare us from applying the Bonferroni correction method). These are called "post hoc" tests.

If you need to know how to run them within SPSS, you should first open the ANCOVA dialog box. Then all the post hoc tests are available by pressing the button labeled "post hoc".
 
#11
I did not get what you exactly mean.
I meant even though, thanks to Dragan, a Tukey test might be less appropriate than the other two tests I mentioned above (although we are not yet sure, because we are not aware of your sample properties), this Tukey test is still a famous one which is incorrectly overused in the literature.

...So to be more conservative, I suggest you to go for Bonferroni and LSD.
 

Dason

Ambassador to the humans
#12
I meant even though, thanks to Dragan, a Tukey test might be less appropriate than the other two tests I mentioned above (although we are not yet sure, because we are not aware of your sample properties), this Tukey test is still a famous one which is incorrectly overused in the literature.

...So to be more conservative, I suggest you to go for Bonferroni and LSD.
Tukey is fine to use in a normal ANOVA setting if you're just looking at pairwise differences. What he was saying was that you can't just directly use it in a ANCOVA type setting.
 
#13
I meant even though, thanks to Dragan, a Tukey test might be less appropriate than the other two tests I mentioned above (although we are not yet sure, because we are not aware of your sample properties), this Tukey test is still a famous one which is incorrectly overused in the literature.

...So to be more conservative, I suggest you to go for Bonferroni and LSD.
Thanks a lot for your detailed answers.

a. Ok. So there is no need to correct for multiple comparison thing in my study.
Actually, this confusion arises as I found an author doing this correction thing in a number of his research studies (published in different peer-reviewed journals).
[If you would like to go through those studies, I will provide you the link.]

b. I know what is post hoc analysis and its various procedures. But I read somewhere that there is no provision for post hoc analysis (in SPSS) for ANCOVA and MANCOVA. The reason given was the same as mentioned by Dragan for Tukey’s approach. That makes me quite worried.
Is post hoc analysis possible for MANCOVA as well as ANCOVA in SPSS?

c. Actually, I have not started collecting data. So, I can’t provide you the sample details.
At this stage, I can only say that I would like to go for pre-post test true experimental design in which 3 groups will be selected at random and treatments/teaching methods will also be assigned to them randomly.
In each of the 3 groups, approximately 80-100 students will be included. (I have calculated this roughly as I already found out on an average the total number of students studying in standard Tenth in a number of schools at my place = 240-300) [students in standard Tenth is my target population].

What other sample properties do you need to answer my query regarding post hoc analysis?
 
#14
a. Ok. So there is no need to correct for multiple comparison thing in my study.
Actually, this confusion arises as I found an author doing this correction thing in a number of his research studies (published in different peer-reviewed journals).
[If you would like to go through those studies, I will provide you the link.]
Sure, would be glad to see those studies. As I mentioned it is controversial, and some authors think that since they have more than one test, they are facing a multiple comparison situation. However, if "the number of tests" was the determining factor, we must reduce the alpha level for almost every study, because in most of the studies, there are more than one test. Besides, this multiple comparison thing says that "we have more than one test, so we might be more likely to obtain some random significant P values." But what are the boundaries to enable us to define the number of tests? Or, how can someone tell if he/she is facing a P value that is randomly less than 0.05? Should we consider the number of tests used in "one" study? Should we consider the number of the tests used on that subject in the literature? Should we count the number of tests being used simultaneously by all the researchers doing statistics exactly at the time that we are as well running our tests? So, counting the number of tests in one study (as that researcher you are giving links to their articles have done) is not necessarily correct, nor can someone really determine which tests to count. So they stick with correcting this issue only in ANOVA-like setups.

b. I know what is post hoc analysis and its various procedures. But I read somewhere that there is no provision for post hoc analysis (in SPSS) for ANCOVA and MANCOVA. The reason given was the same as mentioned by Dragan for Tukey’s approach. That makes me quite worried.
Is post hoc analysis possible for MANCOVA as well as ANCOVA in SPSS?
Yes, post hoc analyses are absolutely available for ANCOVA/MANCOVA in SPSS. I didn't know that we shouldn't use the Tukey one for ANCOVA, but there are plenty of other tests to select. Fortunately the convenient Bonferroni is always there!

c. Actually, I have not started collecting data. So, I can’t provide you the sample details.
At this stage, I can only say that I would like to go for pre-post test true experimental design in which 3 groups will be selected at random and treatments/teaching methods will also be assigned to them randomly.
In each of the 3 groups, approximately 80-100 students will be included. (I have calculated this roughly as I already found out on an average the total number of students studying in standard Tenth in a number of schools at my place = 240-300) [students in standard Tenth is my target population].

What other sample properties do you need to answer my query regarding post hoc analysis?
If you have not sampled and done the experiments yet, be careful that after collecting the data, ANCOVA assumptions might not be met. In that case you have to run some alternative tests for ANCOVA and for the post hocs.

For selecting post hoc analyses, you can use that Bonferroni whatever your sample was (but IF the ANOVA assumptions are met in the first place!).

Since you still have chance to gather more comprehensive data, I suggest you including as many factors which can affect cognitive abilities as possible, such as IQ, the participants' interest in the topic presented, the participants interest in the presenter, the presenters' gender (students with opposite sexes might be more attracted to their teachers), and so many others that are available in the literature (eyesight problems, sleep, the time passed since taking food, female hormonal state, etc.). The more you know about these, the better you can picture what is going on.
 
#16

Dragan

Super Moderator
#17
Tukey is fine to use in a normal ANOVA setting if you're just looking at pairwise differences. What he was saying was that you can't just directly use it in a ANCOVA type setting.
That's right Dason. In fact, SPSS will not allow a user to use any of the post-hoc tests after the user enters a covariate into the model (univariate GLM).
 
#18
victorxstc, Thanks a lot for your answers and timely help and exclusively useful suggestions. I am really really grateful to you.
After data collection, I will definitely and firstly check for ANCOVA assumptions.

Here are the links for the studies in which correction for multiple comparison was being done.

http://onlinelibrary.wiley.com/doi/...nticated=false&deniedAccessCustomisedMessage=


http://www.springerlink.com/content/n7554u60q2h4v267/
You are so welcome :) and thanks for the links :)
 
#19
That's right Dason. In fact, SPSS will not allow a user to use any of the post-hoc tests after the user enters a covariate into the model (univariate GLM).
Interesting! I hadn't tried a MANCOVA with post hocs, so it is interesting to see it doesn't run any post hocs when including any covariates. So what should we do then?
 
#20
That's right Dason. In fact, SPSS will not allow a user to use any of the post-hoc tests after the user enters a covariate into the model (univariate GLM).
Yes exactly, this is exactly what I found while searching for post hoc procedures for ANCOVA/MANCOVA. Then, how to go about for post hoc analysis in case of ANCOVA/MANCOVA?