- Thread starter javedbtk
- Start date
- Tags bonferoni statistical difference

Hi Jave,

n is the number of**tests **you perform.

I think it is better you use the Sidak Correction (similar but more accurate and a bit more power)

n is the number of

I think it is better you use the Sidak Correction (similar but more accurate and a bit more power)

Hi Jave,

What test do you want to run?

How many tests?

For example, if you compare 4 algorithms: A, B, C, D:

If you run the following tests:

A-B (for example t-test to compare A average to B average)

A-C

A-D

B-C

B-D

C-D

In this example case, you run 6 comparisons so n=6.

What test do you want to run?

How many tests?

For example, if you compare 4 algorithms: A, B, C, D:

If you run the following tests:

A-B (for example t-test to compare A average to B average)

A-C

A-D

B-C

B-D

C-D

In this example case, you run 6 comparisons so n=6.

I run A with B, A with C and A with D for dataset 1.

Then A with B, A with C and A with D for dataset 2. Similarly, for 4 datasets. So it means for a particular algorithm A, I will have 3*4=12 experiments, so n will be 12,right?

First I thought, n will be 3, because for each dataset, A is compared with 3 algorithms.

What test do you run for each pair?

If for example, you use a significant level of 0.05 in each test

As the end in each test (pair) you run the allowed probability for type I error is 0.05 but the potential maximum allowed probability in all the test

So the probability not to get a type I error in one test is: (1-0,05)=0.95

The probability not to get a type I error in all the tests is 0.95^12 (but this is the worst case that h0 is correct in all the tests)

together is α'=1 - (1 - 0.05)^12=0.4596

If you use α=1−(1−0.05)^(1/12)=0.004265

The overall α'=0.05

Bonferroni was a bit lazy (just for the joke) and instead of the exact calculation used an approximation 0.05/12=0.00416

He didn't have a computer so it is much easier to calculate manually using the approximation.

This value is a bit small then I calculated.

But when talking about Bonferroni it is generally the "method", I would still use the accurate calculation.

http://www.statskingdom.com/doc_anova.html#sidak