Correction for multiple comparisons applied to Chi2 value rather than p-value

[mousme]

Hi, I am calculating hundreds of Chi2 and I correct for multiple corrections. The script I am using has a threshold at 400 for the Chi2 value beyond which it gives me a p-value of 0. The problem is that it would be really complicated for me to implement the formula for the Chi2 distribution so I was wondering if there was a way to apply a correction to the Chi2 value directly. Thanks!
PS: for each CHi2, I am comparing a observed proportion to a theoretical one.

[victorxstc]

Are you sure your case is really a case of multiple comparison?

Can't you fix this problem using the methods applied to P?

[hlsmith]

Seems like quite a few comparisions. An example of a very crude approach of victorxstc's proposal may be correcting the p-value post-calculation: such as a Bonferroni correction where you divide the level of significance by number of tests; instead you could multiply the p-value for the test by the number of tests (e.g., calculated p-value: 0.0001, with 400 tests, then equals 0.04). You could use a different correction then the Bonferroni, but this one provides a pretty basic example.

[victorxstc]

No I didn't mean the Bonferroni one, necessarily. I meant why doesn't he/she want to fix the problem at the P value level (in general), rather than at the chi-square value level?

[hlsmith]

Yup, I got what you meant and was saying that it may be very easy to implement a p-value correction.

[mousme]

Thank you for your answers. The reason why I would like to correct for the chi2 value and not the p is because the script I use has a threshold of 400 for some reason. This calculation is part of an application which does other things and is written in a particular language in which the script is written. So anyway, beyond 400, it gives me a p-value=0 so any type of correction, Bonferroni or other is not useful. Which is why I was wondering if I could apply a correction 'upstream' as in correct the chi2 values themselves thereby bringing most of them below the 400 threshold. Thanks!

[victorxstc]

I didn't quite understand your comment mousme, but I think by "threshold" you mean a threshold for chi-square values? If so, you can also convert all P values to chi-square values and vice versa, only by knowing the degree of freedom. So they both can act similarly. For example, when DF = 1, an alpha = 0.100 corresponds to chi-square = 2.706; a chi2 = 3.84 corresponds to P = 0.050; and a chi2 = 6.64 corresponds to P = 0.010, etc.

So you can apply the Bonferroni correction at chi-square level. For example, if in a study with 2x2 table, you need a P < 0.05, then you need a chi-square > 3.84. So, if you want to apply the Bonferroni correction, first check that after correcting the multiple comparison problem, which alpha level is desirable. For example, you find that P < 0.01 is now significant (after Bonferroni correction). Then you can calculate and find the chi-square value which corresponds to that corrected alpha (which in this example is 6.64). Therefore, if your chi-square value was greater than 6.64, you are sure that your P value is smaller than 0.01.

All you need is a table of P against chi-square to find the corresponding chi-square value for P value at different degrees of freedom:
http://faculty.southwest.tn.edu/jiwi...robability.htm
http://www.chisquaretable.com/

[hlsmith]

For clarification, it seems that using this method would allow you to redefine the critical values and take into account significance correction. This could be an option.

Was your original issue, that how you are running these tests p-values were getting rounded to 0, so you are unable to correct the p-values themselves, since you do not know the true p-value. Using tables like referenced, you may also be able to look up the actual p-values (unless the number of tests makes this too labor intensive). Also, I have found that some programs (SAS, in particular - you can output values, in which you can get at the infinitesimal values or possibly request outputs to display more decimals. Let us know if these options help or if there still are any issues.

[Mean Joe]

The problem is that it would be really complicated for me to implement the formula for the Chi2 distribution so I was wondering if there was a way to apply a correction to the Chi2 value directly. Thanks!
PS: for each CHi2, I am comparing a observed proportion to a theoretical one.

I'm wondering what formula you're using to correct the Chi2 value? Also, why compare the observed Chi2 to a theoretical one, where's the theoretical one come from?

As for some suggestion for you, do you know that the Chi2 is just ? Maybe you can correct the beta or SE, using whatever formula you are.

[Dason]

As for some suggestion for you, do you know that the Chi2 is just ?

For a regression this may be true but it sounds like they're doing goodness of fit tests instead.