How to correct for comparisons between and within groups?

[arthur.avramiea]

I have two groups - control and treatment - and I'm following response for both groups at 4 timepoints. Data is not normally distributed and I am interested in timepoints where there is a significant difference.

To check whether treatment has effect I look at:
- whether response to treatment at the later 3 timepoints is different from the baseline - non-parametric sign-rank test
- whether response to treatment is different from response to control , at all 4 timepoints - non-parametric rank-summed test

In order to correct for multiple corrections with bonferroni, should I correct for 3 comparisons in the first test, and 4 comparisons in the second test, or should I correct for 7 comparisons in total ?

 

[EdGr]

Obviously, if the data were normal, we would start with a 2-way ANOVA to look for the group by time interaction. If the data are not too badly non-normal, I might do that anyway. It provides a nice support for the follow-up tests. If the interaction is entirely NS, you have no basis for most of the rest.

To compare across time within the treatment group, I would start with a Friedman ANOVA. This would test the overall pattern for significance. I would also do one within the control group, since there may be spontaneous changes.

For the between group comparisons, I would compare at each time on the change from baseline. Are you doing that? I can't tell.

Since some of these are getting at the question from different angles, I think I would test the over-time analyses at alpha 0.05. You are not expecting any change in the control group (or don't much care about it), so you really have only 1 over time ANOVA you care about.

If the Friedman ANOVAs are significant, I would Bonferroni correct the 3 comparisons to baseline at 0.05/3. Some people would compare every time to every other time, which adds to the denominator! But ask yourself how important those comparisons, even to baseline, are. Say the medians are 10 20 30 40 in the treatment group (p = 0.001 by Friedman) and 10 11 10 11, p = 0.9 in the controls. Do you really need to compare each time to baseline? If the pattern is less clear, then you need some comparisons and p-values...

I would use alpha = 0.05/3 for the comparison of groups on the changes from baseline. I would not include baseline in the correction.

take note that getting a p-value that barely makes significance is now starting to be considered very weak evidence. I hope you have p values of 0.001!

 

[Karabiner]

I have two groups - control and treatment - and I'm following response for both groups at 4 timepoints. Data is not normally distributed and I am interested in timepoints where there is a significant difference.

Whether the data within the groups are sampled from normally distributed populations
is not an important issue if sample size is not too small. Description of sample size is
missing here, unfortunately.

In order to correct for multiple corrections with bonferroni, should I correct for 3 comparisons in the first test, and 4 comparisons in the second test, or should I correct for 7 comparisons in total ?

You could perform global tests (repeated-measures ANOVA or Friedman test),
and if they turn out "statistically significant", then you could consider pairwise
comparisons without correction. Why decrease statistical power if it already
clear that there are statistically significant differences.

With kind regards

Karabiner