I received a review of my manuscript in which one of the three reviewers advised that I will need to apply a Bonferroni correction due to multiple comparisons. However, based on the examples I have reviewed, I am not certain that a Bonferroni correction is needed in this case. Here is the description of my data and current analysis:
I collected data from a set of 10 human subjects. Each subject had their data recorded on 5 different dates during 5 different testing sessions. Each subject trained an artificial intelligence program to perform a specified task, and during each session, 3 different methods of training the AI program were used (the same 3 methods were used for all subjects and across all testing sessions; the human subjects provided the data for one of the methods, whereas the other 2 methods used algorithms to generate the associated data). For each session, I am performing a Kruskal-Wallis ANOVA analysis with post hoc multiple comparison to compare, using 3 different outcome metrics, how successfully the set of 10 subjects trained the AI program to perform the task using the 3 tested methods, with the comparison between each pair of methods being reported as either significantly different or not based on a critical p-value of 0.05.
In case it is relevant, the 5 sessions in which each subject participated "built upon" each other; e.g. the AI program that was trained in Session 1 was used as the starting point for Session 2 for the same subject, so that the AI program's performance gradually improved over the 5 sessions performed by each human subject.
I am wondering whether a Bonferroni correction is really necessary, since for each of the 5 sessions, the data was collected separately from the data of the other sessions, so that performing parallel analyses on the 5 sessions might not necessitate a correction of the p-value in the same way that performing numerous comparisons on the *same* dataset would.
Or, since I am comparing 3 different outcome measures for the 3 conditions being compared for each session, would I need a Bonferroni correction factor of 3 for the 3 outcome measures being used?
Thanks in advance for any advice you can provide.
I collected data from a set of 10 human subjects. Each subject had their data recorded on 5 different dates during 5 different testing sessions. Each subject trained an artificial intelligence program to perform a specified task, and during each session, 3 different methods of training the AI program were used (the same 3 methods were used for all subjects and across all testing sessions; the human subjects provided the data for one of the methods, whereas the other 2 methods used algorithms to generate the associated data). For each session, I am performing a Kruskal-Wallis ANOVA analysis with post hoc multiple comparison to compare, using 3 different outcome metrics, how successfully the set of 10 subjects trained the AI program to perform the task using the 3 tested methods, with the comparison between each pair of methods being reported as either significantly different or not based on a critical p-value of 0.05.
In case it is relevant, the 5 sessions in which each subject participated "built upon" each other; e.g. the AI program that was trained in Session 1 was used as the starting point for Session 2 for the same subject, so that the AI program's performance gradually improved over the 5 sessions performed by each human subject.
I am wondering whether a Bonferroni correction is really necessary, since for each of the 5 sessions, the data was collected separately from the data of the other sessions, so that performing parallel analyses on the 5 sessions might not necessitate a correction of the p-value in the same way that performing numerous comparisons on the *same* dataset would.
Or, since I am comparing 3 different outcome measures for the 3 conditions being compared for each session, would I need a Bonferroni correction factor of 3 for the 3 outcome measures being used?
Thanks in advance for any advice you can provide.