Probing Interactions and High and Low values

#1
Hi, I am running some analyses to probe a significant three-way interaction that I obtained with three continuous variables (stress, alcohol use, and insecure attachment). To probe the interaction I chose 1SD above and below my mean for alcohol use and attachment. At low levels of alcohol use and at low levels of attachment, the relationship between stress and aggression is marginally significant (p = .06). (For all other combinations of alcohol use and attachment, the relationship between stress and aggression was not significant).

For curiosity sake, I then probed the interaction at 2DS above and below the mean for alcohol use and attachment. The relationship between stress and aggression was significant (p < .05). Is it okay to probe the interaction at more extreme values (i.e., 2 standard deviations versus one standard deviation) and what implications does this have for my findings? Would this make my findings suspect? Any information would be most welcome! Thanks!
 

Jake

Cookie Scientist
#2
I think it starts to look pretty (data-)fishy when you start doing that. I would not recommend reporting such a test. I think your p of .06 at a more reasonable and conventional level of the moderating variable is perfectly respectable. As Bob Rosenthal noted, "surely, God loves the .06 nearly as much as the .05."
 
#3
Okay. That makes sense. Do you think it is strange though that although my 3-way interaction was significant, upon probing, the one combination that was driving the significant interaction (i.e., the association between stress and aggression at low levels of alcohol use and low levels of attachment security) is only marginally significant? Thanks so much for all of your help!
 

Jake

Cookie Scientist
#4
No, not strange really. A significant higher-order interaction does not necessarily imply that any of the lower-order terms are significant. Your data is sort of proof of this in itself, but to get a more intuitive understanding of how this can be so, consider a hypothetical factorial experiment where I have 3 predictors A, B, and C, and where I find a significant A*B*C interaction. One way to interpret this interaction is that the 2-way A*B interaction is moderated by C. So I break down the A*B interaction into two plots where C is either high or low and this is what I see:

Although the nature of the two way interaction changes sign between the two plots, and thus they are very different from each other, both of the 2-way interaction effects in each graph are clearly very small. This is a situation where I might have a significant 3-way interaction even though neither of the component 2-way interactions are significant. The key difference here is that each two-way interaction tests whether the interaction effect is different from 0, while the 3-way interaction tests whether the two 2-way interactions are different from each other. See the difference?