Box's Test sig .000 - now what?

#1
I'm running a manova (teaching myself along the way) and I can't seem to get a handle on something - I'm not sure if I'm having a sample size problem or a cases/variance problem and I'm a little frustrated with conflicting opinions about how to address the issue (or if it even needs to be addressed). I'll give as much information as I can and if anyone has some advice, I'd appreciate it.

I have 1 categorical IV with 4 values ranging from 1-4.

I have 5 continuous DVs (representing a single phenomenon) that range from 1.00 to 5.00 (with both decimal places used i.e. dv1-2.33, dv2-3.14, dv3-4.90, dv4-2.00, dv5-1.28).

I know sample size can be an issue - my sample is small: 227.

My count for all DVs is 227, so there's nothing there to cause issue.

I've been told that if Box's is significant, I have to throw it all out... that seems a little extreme. There has to be a fix.

I've also been told that Box's is too sensitive and that I shouldn't use it, but I am suspicious of advice that isn't backed by a good reason.

I've been told that because I'm using 2 decimal places with a small sample, Box's will be .000 no matter what I do to correct the alpha.

I've been told that .000 is fine for what I'm doing as long as I use Pillai instead of Wilks - again, not told WHY, so I'm suspicious.

I've also been told that because my question is simply "is there an effect on the DV from the IV, Box's doesn't matter - that doesn't make much sense. My understanding is that Box's is a test for equality of variance of the DV across groups.

Basically, I've been given a lot of advice and not much direction.

My simple question is this: in my case, what do I do (if anything) to address the sig of Box's test? What's my next step?

Thanks for your time. Sincerely.
 
#3
@mjgray,

You have not demanded an ASAP or written any other non-understandable abbreviations or written in such a way that it is hard to read, so I guess that there is no need to avoid answering. The “please” doesn't matter at all (sorry :) ), but your question is a little bit interesting.

I would avoid the multivariate anova and run it like univariate anova for each dependent variable.

Like:
DV1 vs IV
DV2 vs IV

(That is what people do anyway, and it is very seldom that they just have one single dependent variable. They have lots or them and they run them one-at-a-time. That is completely satisfactory in my view.)

In my experience if you have one dependent variable and the IV is significant then it will also be significant in manova. If it is not significant in any univariate anova it will not be significant in manova either. So to me manova seems very attractive in theory but not very important in practice.

If you have many IV:s you of course include them in the model and have one DV but it is still only a univariate anova. It is the number of dependent variables that makes it a univariate analysis. A multiple regression model with several x-variables (and one dependent variable) is a univariate model.

A sample size of 227 (and 50, 60 in each group) is quite large so that large sample properties will kick in, eg. that the estimates gets approximately normal. Hmm, they would be that anyway since this is anova and that is assumed to be normally distributed so that linear combinations that created the estimates will be normally distributed.

But if standard deviation is large and you want to “discover” a small difference the sample size mights still not be large enough.

Check that the residuals a reasonably normally distributed (and at least no severe outlier) and that the residual variance in each group is about the same. (But anova is robust to violations of that (but not to outliers) so it doesn't matter that much.)

If you have had a previous thread about this study then it is good if you link to that thread.

@mjgray. Do you know what is “GCSE education” and “KS3 education”? I have no idea and would like to know.
 
#4
Thank you for your response.

I have no idea what GCSE or KS3 are, but I looked them up real quick. Apparently, they are a thing: http://en.wikipedia.org/wiki/General_Certificate_of_Secondary_Education and apparently, there was a bit of controversy this past year: http://www.bbc.co.uk/news/education-20664793 (sounds like a good topic for a debate round).

Digressing...

The reason I chose manova was because the DVs are interrelated. My study measures an assumed predictive IV against several interrelated DVs, much the same way a study might test private/public school (as the DV) against math, science, and history test scores to see the effect on "overall educational success". The theoretical foundation of my study strongly supports the idea that there will be a specific effect, so I wasn't surprised when the sig was .000 - I was happy that it was that severe, but not too surprised. Then I ran the Box's test and all of my hopes were shattered.

SPSS renders a table for individual anova when you run the manova, but I suspect running them separately will generate slightly different nuances (though, I honestly do not know why). Those tables show sig < .05 relationships between the IV and some (but not all) of the DVs, but I think (perhaps wrongly?) that I would have to justify NOT running a manova with IVs that, when combined, represent a single phenomenon. Or... well, actually this may be the answer: while the IV does indicate a significant effect on some of the DVs, the IV is not seen as a good predictor for the overall phenomenon represented by the DVs. I think I might be able to go with that, though I'd like to look over an existing study where a similar conclusion was drawn to help me situate the idea.

My IV has 4 groups (or values). 3 of the 4 are very important to the study, but removal of the 4th (which only has 19) can easily be justified - it's just not that theoretically interesting. One of the remaining 3 only has 28 and the rest of my sample is divided pretty evenly between the last 2 important groups. I've been considering whether or not I can justify targeting a new (similar) sample that is likely to bolster the category with only 28 to even everything out, but I'm not sure I'm comfortable with that. It seems ethical and I could just hollar "FOR SCIENCE!", but that probably won't get me very far. I need a precedent that I have yet to discover to justify such action.

All that said, I suppose that if i can get around the manova all together, that's the solution... but, as I said, I feel like I would need to justify not running the test.

Thanks again for your response! If you or any others have anymore input on the matter or direction for the best method for dealing with those nasty outliers, it would be greatly appreciated!

I'm currently scouring the internet for the best method to identify and remove outliers, while I wait for Kerlinger's "Foundations of Behavioral Research" to come in.
 
#5
Outliers!?

You never said anything about any outlier before. You only said that the variance-covariance matrix were not the same.

Why don't you tell us what your DV is and and give a few numeric examples like 5, 6 values and show a histogram and QQ-plot for the univariate residuals.
 
#6
Hey - sorry I haven't responded till now. I've been learning.

So, outliers aren't an issue. I thought they might be, but I was wrong.

I took your advice and dropped the whole manova thing. Separate anova are working just fine. All of my tests pass Levene's test and I find a significant effect of the IV on 2 or the 5 DV. "post-hoc" (what do you call post-hoc when you were planning to do it from the beginning?) give me direction for the effect and I have plenty to talk about.

However... I am interested in accounting for a covariate to see what happens. Trouble is, I'm not sure what the real difference between a One-way anova and univariate analysis is. I know that the results come out the same, so the analysis of variance must be basically doing the same thing and the ONLY reason I need to run my data through univariate in addition to anova is because anova doesn't allow me to designate a covariate. I already know that the covariate does not effect significance, but I'd like to report that as well. The trouble is, where univariate is concerned, I don't know what I don't know... and I might need to know it (whatever it is) when defending.

Does that make sense?
 
#7
I'm not sure what the real difference between a One-way anova and univariate analysis is.
A univariate analysis is where there is just one dependent variable and there could be one or many explanatory variables. Formally when someone is doing a multiple regression analysis it is a univariate analysis, although it is often called a “multivariate analysis” but that is formally incorrect. (But people tend to be quite forgiving in this matter.)

A one-way anova is a (univariate) analysis of variance where there is just one explanatory factor with two or more levels, like “young”, “middle” and “old”.

Don't take the names from spss.That is a very old software package and they have kept on adding new features and sometimes the same thing can be done several procedures. You can run t-test in many procedures.

A post hoc test is still called a post hoc test even if it is planned from the beginning. It is just a name and used like that for historical reasons.

The trouble is, where univariate is concerned, I don't know what I don't know... and I might need to know it
It is just to continue a balanced study of statistics and your own subject matter. You can't know everything and nobody expects you to.
 
#8
Thanks so much.

If anyone feels like taking the time, I'd really like to know why reporting df is important. Seems like relatively useless information that can be figured out in about 2 seconds by the reader. And I can't think of any good reason why they'd want to know...
 
#9
It is not important with degrees of freedom.

What is important is the estimated effect and if it is significant (or the length of the confidence interval).

But at the time when people looked at tables to figure out if if it was significant or not, they needed to know the degrees of freedom to pick out the correct critical value. You don't need to report it.
 
#10
That's what I was thinking. I can see that there was a time when it was necessary... So, why is my committee telling me to report it? I mean, I'll do whatever they say, but I don't get it. I'm being told on one side "don't include things that aren't necessary" and then being directed to include what is probably the most unnecessary bit of information. Besides it being unnecessary, it's not as if I really need to report it. If people know my sample size and number of cases, they can do the math. It's literally 227 minus 4. How hard is that?

At the end of the day, I'll just report whatever they say to report... I just can't help being a little annoyed that I'm not being given a good reason why...
 

Dason

Ambassador to the humans
#11
Well I agree that it's easy enough to calculate ... I don't think it's completely useless though. When I was doing consulting people would bring me some analysis that they had run and after getting a feel for the experiment I would look at their model and the degrees of freedom and more often than you would hope the degrees of freedom wouldn't match up with what they should be. So it's a useful piece of information for somebody that is trying to verify if what they imagine the model actually is matches what you actually did.
 
#12
That makes sense. At least there's a reason now... it absolutely doesn't apply in my case, but, taking that into account, I can see why it's a norm.
 

Dason

Ambassador to the humans
#13
That makes sense. At least there's a reason now... it absolutely doesn't apply in my case, but, taking that into account, I can see why it's a norm.
Alright. I'll be honest and say that I didn't really read much of the thread but I saw you asking why somebody might want the degrees of freedom to be reported and I thought I'd share an anecdote which might shed some light on why it could be useful.
 
#14
Wasn't really on topic with the thread. Thanks for the response though. I wasn't thinking of other cases (like the one you presented) where it may be needed. I suppose there are things we do, even if we don't need to in each specific case, on the off-chance that it will be needed. It's a better answer than "just because", ya know?