2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase power?!

#1
I am in a very frustrating place, so I would appreciate any help/brainstorming that you can muster!

I am conducting analyses related to a quasi-experimental study I have completed. Two age groups (Young, Old) were randomly assigned to three condition (Exp1, Exp2, & Control).

I have completed a 2 (age) x 3 (condition) factorial anova, and I predicted that there would be a significant interaction. However, the interaction is marginally significant (p = 08). I am so frustrated, because I would like to proceed to post-hoc analyses. It is particularly frustrating because if I conduct a 2 (age) x 2 (exp conditions) factorial anova, everything turns out beautiful. Somehow including the control condition messes everything up!

My question is: What should I do?
1) Should I proceed to post-hoc with only a marginally significant effect? Will that raise questions later on?
2) Is there some way that I can structure my analyses so that increase power and push the probability over the .05 threshold?

Feel free to offer suggested reading or example articles, and will do the research to save you time explaining out the details.

Thank you!
 

noetsi

No cake for spunky
#2
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

I don't understand why you are frustrated that interaction did not occur (at the .05 level). Most would be delighted, since without interaction it is easier to explain main effects. I am also confused why you can not go to post-hoc tests because you did not find interaction. If anything interaction would make interpretation of post hoc tests more difficult.

When you say "interaction" do you actually mean that the omnibus F test was not signficant? Interaction has nothing to do with whether the omnibus test is significant or not. Nor does interaction prevent you from going to the post hoc test (it makes their interpretation more doubtful, but since you did not find it that would not be a problem)?

1) If you mean that the omnibus F test was not signficant then some feel you should not go to the post hoc test, and some say you can. Whether it raises questions depends on who reviews your paper. I think the trend is to downplay the signficance of the omnibus F test, but again there is disagreement on this issue. If you think this may be a problem one possible solution is to increase your power.

2) The key question then is, what is your power? The easiest way to increase your power is to increase your sample size. I believe the number of levels of your variables also influence power, although I will leave that to others to comment on. ANCOVA may also help increase power, depending on the nature of your data.

I think you should clarify if the problem is really interaction or the omnibus f test.
 
#3
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

If your interaction is not significant, you should not continue with post-hoc tests. The significance of the interaction gives you the "green light" to do such tests, and as you have no significance then you should not continue (that is, after all, why there is a criteria for significance, as arbitrary as the .05 level is).

Performing your analysis without the control condition could be an avenue to explore, but then that begs the question if the control condition is not important enough to include in the analysis, then why was it included in the experimental design? At the end of the day, you have to resort to scientific honesty and present the results that you intended to analyse a priori. If that means your interaction is not significant, then so be it.

Regarding power, again the best option is probably to run a few more subjects. If the interaction effect is real, it should come through after a few more subjects. Although most psychologists do this (I have done it at a reviewer's request), I am quite uncomfortable with it. Power (and thus sample size) should be decided a priori, and you will do well to stick to the rigour of only running the subjects you wanted to run. Based on this, I would suggest reporting what you have and not performing post-hoc analysis.

Regarding a different set of analyses to increase your power, I am not sure what to suggest without seeing more of what your experiment was. For example, my dependent variables of choice are reaction time and percentage error. I do data trimming on the RTs (i.e. get rid of very slow RTs), and I also exclude participants who make more than 10% errors. Could you do something similar? Again, try and be as honest as you can, and only remove subjects with good reason.

Best of luck! And remember, null results are often more interesting than significant results!!

Cheers,

Jim.
 

Dason

Ambassador to the humans
#4
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

If your interaction is not significant, you should not continue with post-hoc tests. The significance of the interaction gives you the "green light" to do such tests
I don't quite understand your logic here. Why are you worried about using the interaction as an indicator of the appropriateness for post-hoc tests? What if you originally hypothesized that there wouldn't be an interaction? Do you still say you shouldn't do a post hoc test?

I'm concerned about the appropriateness of the ANOVA if including the control takes away the significant result. Have you checked the equality of variance assumption? Have you checked normality of the residuals?
 

noetsi

No cake for spunky
#5
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

In honesty I am baffled why you need interaction to go onto a post hoc test. Never seen that in an ANOVA text or class :)
 
#6
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

Just to clarify, I hypothesized an age x condition interaction (I did not hypothesize main effects). Therefore, I wanted the interaction to be significant so that I could follow-up on it (aka, tease apart the interaction). Unfortunately, the p value for the initial interaction term was marginally significant (.08) and I therefore question whether or not I have the rational to proceed to tease apart marginally significant interaction. If I move forward with the post-hoc (or if I run the analyses without the control condition, aka: a 2x2 design), I can see that there are age effects for one of the experimental conditions. However, I can't just chuck away the control condition just because it impacts the overall F value of the initial 2x3 ANOVA. I hope this brings clarity to my original question.
 

Dason

Ambassador to the humans
#7
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

Did you check the validity of the assumption of equal variances?
 
#9
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

In honesty I am baffled why you need interaction to go onto a post hoc test. Never seen that in an ANOVA text or class :)
To do post hoc test on the interaction, of course you need the interaction to be significant (this is what I was referring to). Post hoc tests on the main effects with no interaction? No problem! I have never seen in any academic article an author explore an interaction using post hoc tests when the interaction was not significant. Why bother with the ANOVA in the first place if this is your strategy?
 
#10
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

Unfortunately, the p value for the initial interaction term was marginally significant (.08) and I therefore question whether or not I have the rational to proceed to tease apart marginally significant interaction.
I still stick by my guns and suggest that you should not be exploring the interaction using post hoc tests when your ANOVA interaction is not significant.
 

Jake

Cookie Scientist
#11
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

Ah, the tyranny of the omnibus test lives on! It really is so much nonsense. The 2-df omnibus test on the interaction tells us: "there is somehow some interaction between some combination of the groups." Is that a question that we really care about? Jamie, did you conduct your experiment to settle the answer to this vague and unhelpful question? I presume not. So why on earth should we only investigate the questions that we actually care about in the data, the questions that motivated the experiment in the first place, conditionally based on the answer to this omnibus question?

This kind of reasoning made good sense in the agricultural context in which ANOVA was originally applied. In such an experiment you would throw a bunch of combinations of plants and treatments together and basically say "let's see what happens!" So yeah, under that approach to experimentation, these kinds of strategies for protecting against alpha inflation are quite sensible. But that strategy doesn't seem to make much sense outside of that context. I submit that you ought not to consider yourself strictly bound by this agricultural convention.

I detect a sentiment that the entire point of ANOVA is to protect against the alpha inflation associated with doing a sequence of t-tests. I hardly know how to reply to such an idea except to point out that ANOVA is manifestly not equivalent to simply doing a series of t-tests while basically maintaining the nominal alpha level. We perform ANOVAs for a variety of reasons, including increased power, the ability to ask interesting substantive questions such as quadratic differences, the ability to "control for" factors, etc. An ANOVA is certainly not just an alpha inflation adjustment.
 
#12
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

Hi Jake,

Although I don't disagree with what you say, until the field of psychology changes its stance on statistics, you will not get a paper accepted in a reputable journal where you have performed post-hoc tests on an interaction when the interaction in the ANOVA was not significant. That much, I think, is quite clear. So, I still stand by my suggestion (in a psychology stats forum) that post-hoc tests on the interaction should not be conducted, unless for purely exploratory purposes.

But, if we are talking about the ridiculousness of some stats, I think the focus too should be on .05 as the criteria for significance! :)

Cheers,

Jim.
 

jpkelley

TS Contributor
#13
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

And you definitely wouldn't consider designing an experiment like RA Fisher did when he first applied ANOVA...lest you want to get reamed by the "pseudoreplicationists."

So, one thing that should be asked is how many times do individual (i.e. pairwise) post-hoc tests come out as significant (after correction for multiple comparisons, blah, blah) when the interaction is NOT significant. Couldn't the convention of stopping your exploration simply be a suggestion of convenience rather than a tool for strong inference?

Secondarily, I do agree that there needs to be a balance between what is ideal/perfect and what is going to be understood and accepted (i.e. practical idealism).
 
Last edited:

noetsi

No cake for spunky
#14
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

To do post hoc test on the interaction, of course you need the interaction to be significant (this is what I was referring to). Post hoc tests on the main effects with no interaction? No problem! I have never seen in any academic article an author explore an interaction using post hoc tests when the interaction was not significant. Why bother with the ANOVA in the first place if this is your strategy?
The analysis that I have seen use the omnibus F test not a hypothesis about a specific set of means to go on to post hoc test. If you have a hypothesis about a specific set of means why would you use a post hoc test rather than calculate a contrast (which has more power)?

Under the rule of marginality you should never have an interaction term in a model without main effects.
 

Jake

Cookie Scientist
#15
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

Er... not familiar with this "rule of marginality" but it sounds quite silly if I understand it correctly. Consider the following made up cell means:

There are two factors, let's call them "color" (black vs. white) and "side" (left vs. right). There is no simple effect of color, that is, the black bars are not higher or lower than the white bars on average across side. Likewise, there is no simple effect of side, that is, the left bars are not higher or lower than the right bars on average across color. But clearly there is a substantial crossover interaction here. Are you suggesting that this interaction should not be possible??
 

noetsi

No cake for spunky
#16
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

From John Fox

"As a corollary to this principle, it does not generally make sense to specify and fit models that include interaction regressors, but that omit main effects that are marginal to them. This is not to say that such models - which violate the pinciple of marginality - are uniterpretable: They are not rather, broadly applicable."

He is discussing it in the context of regression, but similar comments can be found in ANOVA text. The first I ever read said flatly never to exclude main effects when analyzing interactions :)
 

Jake

Cookie Scientist
#17
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

I see - I misunderstood you. I thought you were saying something along the lines of "in a model that includes an interaction term, you cannot have a significant interaction without at least one simple effect being significant." Which clearly is nonsense. But now I see that that's not what you were saying at all. :)

Indeed, I was taught rather simply: "don't put in an interaction term without including its component terms... just don't do it!!" So I never even took the time to learn what the proper interpretation of such a model would be. But I have to admit that I am curious. I wonder if someone here more familiar with the issue than myself could give us a quick illustration of what the proper interpretation would look like for the coefficients of a model such as Y = b0 + b1X + b2X*Z ?
 

spunky

Can't make spagetti
#18
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

I wonder if someone here more familiar with the issue than myself could give us a quick illustration of what the proper interpretation would look like for the coefficients of a model such as Y = b0 + b1X + b2X*Z ?
do you mean here that Z is the usual multiplicative product of X1 and X2 or is Z just multiplying X2?
 

noetsi

No cake for spunky
#19
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

I see - I misunderstood you. I thought you were saying something along the lines of "in a model that includes an interaction term, you cannot have a significant interaction without at least one simple effect being significant." Which clearly is nonsense. But now I see that that's not what you were saying at all. :)

Indeed, I was taught rather simply: "don't put in an interaction term without including its component terms... just don't do it!!" So I never even took the time to learn what the proper interpretation of such a model would be. But I have to admit that I am curious. I wonder if someone here more familiar with the issue than myself could give us a quick illustration of what the proper interpretation would look like for the coefficients of a model such as Y = b0 + b1X + b2X*Z ?
There is disagrement on this point (note that normally you have to have two main effects to have an interaction - which I don't see in your example, but I assume is there). Some say it does not really matter if the main effects are signficant or not - don't interpret them with signficant interaction (be that regression or ANOVA). Others say that you can interpret them if the interaction is not disordinal. That is when the levels of the categorical variable don't shift their relative order, they simply are not parallel to each other at different levels of the other IV. For instance if females always have higher results on the DV than male (but the gap between them varies at different levels of another IV) you can interpret main effect despite the interaction.

Others say just do simple effects.
 

Dason

Ambassador to the humans
#20
Re: 2X3 Anova: Interac is marg sig (p = .08) do I proceed to post hoc or increase pow

There is disagrement on this point (note that normally you have to have two main effects to have an interaction - which I don't see in your example, but I assume is there).
I'm pretty sure he was asking specifically about the case where you don't have both main effects in the model so assuming that the other main effect is there would be the wrong thing to do.

My main question to Jake would be are we assuming that Z is categorical or continuous?