Did I run the right tests?

#1
Greetings,

I hope someone can take a look at the tests I ran and let me know if it seems appropriate, or if I'm off track. BTW - I'm using JASP for my analyses (just based on a google search for something I could use).

My research project is a group design. Participants complete a data entry task under four different conditions (plus a baseline). The dependent variable is the number of records completed. There are two groups with the conditions counterbalanced, for example:

ABCDE
ADEBC

I ran a repeated measures ANOVA on the conditions, regardless of the order in which they were introduced (ABCDE). I used the group assignment as a between-subjects factor. The results are below (sorry, they don't copy and paste very well). The within subjects had a p<.001, and the between subjects had a p=.162. The within subjects did not provide each comparison, so I ran a paired samples t-test to get a better look at how each condition performed against the others.

Does that sound reasonable? Would it also make sense to run a repeated measures ANOVA for trials 1, 2, 3, 4, 5 - even though the participants experienced the conditions in a different order?

Thank you so much for any assistance you can provide.

Within Subjects Effects
Cases Sum of Squares df Mean Square F p η²
RM Factor 1 12274.520 4 3068.630 66.287 < .001 0.343
RM Factor 1 ✻ V2 3468.806 4 867.201 18.733 < .001 0.097
Residuals 3518.280 76 46.293

Note. Type III Sum of Squares

Between Subjects Effects
Cases Sum of Squares df Mean Square F p η²
V2 1651.142 1 1651.142 2.113 0.162 0.046
Residuals 14848.820 19 781.517

Note. Type III Sum of Squares

Paired Samples T-Test
t df p Cohen's d
CoopS - CompS -4.784 20 < .001 -1.044
CoopF - CompF -6.783 20 < .001 -1.480
CoopS - CoopF -0.572 20 0.574 -0.125
CompS - CompF -1.692 20 0.106 -0.369

Note. Student's t-test.
 
#2
I used the group assignment as a between-subjects factor.
Seems fishy, I think this should be one-way repeated measures, once you discount the washout/period effects. What was the 'within subjects' effect then?
 
#3
Hmm, I suppose I just wanted to be certain whether or not the order that the groups experienced the conditions impacted the effects of the conditions. I can run again without the between-subjects factor.

Thanks for the response!
 
#4
oh I see. by group assignment you mean
ABCDE
ADEBC

?

That is probably OK, but I am thinking drop the 'Group' effect and refit as one-way repeated measures, with 'a','b','c','d','e' as the within subjects once you are satisfied with no group effect. the paired t-tests are good to go. I don't remember how to analyze the order effects in this situation, it may not be possible under this design.
 

Karabiner

TS Contributor
#6
Does that sound reasonable? Would it also make sense to run a repeated measures ANOVA for trials 1, 2, 3, 4, 5 - even though the participants experienced the conditions in a different order?
The test of the main effect of "group" wasn't what you were really interested in
("does one group show a higher level of the dependent variable across conditions"),
but rather it was the interaction ("is the difference between A and B dependent
on order")?

I also wouldn't rely on non-signficance, but also on the magnitude of the
interaction effect in the sample. After all, you aren't interested in the effects
of "order" for theoretical reasons, but just to rule out possible bias caused by
the experimental design. With large samples, even irrelevant effects would
become "statistically significant". With small samples, even a relevant impact
of order could maybe become nmon-signficant.

If you cannot rule out a relevant impact of "order", you could make pairwise
comparisons by using smaller versions of your "mixed" ANOVA (i.e., instead
of all 5 levels of your repeted-measures factor you include only 2 in each
analysis).

With kind regard

Karabiner