This question concerns the chance of Type I error in hypothesis testing with multiple statistical tests. The data come from a series of experiments. To make it easier to describe clearly, we could consider the experiment as having 3 parameters that could be varied. We could call these A, B, and C. A has five levels, B has four levels, and C has three levels. Some combinations of A, B, and C don't go together, so when we remove those we have 42 distinct combinations left.
For each of the 42 combinations, we test the combination of treatment parameters on each of seven subjects to yield a single real number as a result. Per the professor's specification, it is the ratio of the alternative treatment result to the standard treatment result. We separately compute the mean for each of these 42 treatments and test the hypothesis that each of the 42 mean values is equal to one, i.e. standard treatment is equivalent to alternative treatment. Is the Bonferroni adjustment required here even though each of the 42 t-tests is considered independent of the others? (The trials are still each being done on the same seven subjects at different times.)
In addition, the professor says we must also stratify according to the five levels of A to perform pairwise t-tests between those levels. (He did not specify how to handle levels of B and C.) The null hypothesis is that the means are equal. Statistics professors have rightly cautioned against "data dredging" and performing too many statistical tests on the same set of data to try to find something significant. If it were the pairwise t-tests only, perhaps the Bonferroni adjustment would help, but performing one-sample t-tests then performing pairwise t-tests on the same data is troubling.
Just 2 questions:
1. Is some other test better than pairwise t-test here?
2. Is some appropriate correction possible for these different kinds of test on the same data?
The search for answers gave these leads, none viable so far:
This post applies to part of the problem mentioned above, but no-one has answered the post so far:
http://www.talkstats.com/showthread.php/5707-One-sample-t-tests-and-bonferroni
This other post may apply, but no-one gave a definitive answer so far:
http://www.talkstats.com/showthread...ts?highlight=familywise+error+multiple+t-test
This similar one went unanswered as well:
http://www.talkstats.com/showthread...sures-with-Bonferroni-and-t-tests-(no-t-value)
Thank you for your advice!
For each of the 42 combinations, we test the combination of treatment parameters on each of seven subjects to yield a single real number as a result. Per the professor's specification, it is the ratio of the alternative treatment result to the standard treatment result. We separately compute the mean for each of these 42 treatments and test the hypothesis that each of the 42 mean values is equal to one, i.e. standard treatment is equivalent to alternative treatment. Is the Bonferroni adjustment required here even though each of the 42 t-tests is considered independent of the others? (The trials are still each being done on the same seven subjects at different times.)
In addition, the professor says we must also stratify according to the five levels of A to perform pairwise t-tests between those levels. (He did not specify how to handle levels of B and C.) The null hypothesis is that the means are equal. Statistics professors have rightly cautioned against "data dredging" and performing too many statistical tests on the same set of data to try to find something significant. If it were the pairwise t-tests only, perhaps the Bonferroni adjustment would help, but performing one-sample t-tests then performing pairwise t-tests on the same data is troubling.
Just 2 questions:
1. Is some other test better than pairwise t-test here?
2. Is some appropriate correction possible for these different kinds of test on the same data?
The search for answers gave these leads, none viable so far:
This post applies to part of the problem mentioned above, but no-one has answered the post so far:
http://www.talkstats.com/showthread.php/5707-One-sample-t-tests-and-bonferroni
This other post may apply, but no-one gave a definitive answer so far:
http://www.talkstats.com/showthread...ts?highlight=familywise+error+multiple+t-test
This similar one went unanswered as well:
http://www.talkstats.com/showthread...sures-with-Bonferroni-and-t-tests-(no-t-value)
Thank you for your advice!
) The 42 one-sample t-tests show whether each combination of pulse width, frequency, and current level is better than the control pulse. Like most biological examples, the muscle is nonlinear in its response, so the results are not what you would expect by linear thinking. That is, doubling the electric current does not, as a rule, double the force output. Given that a parametric math model for this tissue is not yet available, the statistical tests help to show the truth so the math modelers can work on developing a coupled differential equations model for this particular gastrointestinal tissue.