# Thread: bonferroni correction in multivariate regression

1. ## Re: bonferroni correction in multivariate regression

Originally Posted by rogojel
If I understand correctly what CowboyBear says then I would disagree - the assumption that most of the coefficients are exactly zero ( that is they have no effect on my DV) seems to me to be reasonable.That seems to me to be equivalent to Occams razor or the paucity of effects principle from the DoE crowd
Interesting, I've never heard of the paucity of effects principle. Could you expand?

Occam's razor says we should prefer the simpler of two explanations that are equally good at explaining the same set of observations. It is not a guarantee that the world itself is simple, so I don't think it's relevant here. You could say that the idea that parameters tend often to be zero is a theory, but that theory would by definition not be as good at explaining actual observations as a theory allowing parameters to vary, so again the razor is of limited relevance.

If you'd like an empirical demonstration of my point, see this article: Empirical statistics: IV. Illustrating Meehl's sixth law of soft psychology: everything correlates with everything

The authors take a haphazard grab-bag of 135 educational and biographical variables, and examine their association in 2058 subjects. Despite being conceptually unrelated in many cases, each variable had a significant correlation with about 41% of the other variables. (Not a regression approach, but you get the idea).

2. ## The Following User Says Thank You to CowboyBear For This Useful Post:

rogojel (11-27-2014)

3. ## Re: bonferroni correction in multivariate regression

Originally Posted by rogojel
If I understand correctly what CowboyBear says then I would disagree - the assumption that most of the coefficients are exactly zero ( that is they have no effect on my DV) seems to me to be reasonable.That seems to me to be equivalent to Occams razor or the paucity of effects principle from the DoE crowd.
Well remember how SSTs are applied: we either reject the null hypothesis or we fail to reject it. Note that failing to reject a null hypothesis does not equate to accepting the null hypothesis.

4. ## Re: bonferroni correction in multivariate regression

hi,
the paucity of effects means thatbwhen we design a screening experiment we can assume that most of the factors will be inert , that is they will have no effect on the studied outcome. E.g if we have 10 factors then we can investigate 1024 different effects for main effects and various interactions between the factors - and we very definitely expect most of those to have a coefficient that is exactly zero.

I see Occams razor in a bit less restrictive way. My latin fails me , but IIRC the original statement says something like you should not multiply the causes without need. That would fit into,the view, that one should aim for the model that has the least number of parameters without a serious deterioration of model quality, this latter being defined in some objective way e.g. predictive RMSE or such.

@injektilo - I do not see the practical difference. If I can not reject the null hypothesis that would mean, in practice,mthat I have no basis, for example, to request a new investment in a plant for some machine that will control that factor. If I can prove the null hypothesis false, I have the needed arguments to request such an investment.

regards
rogojel

5. ## Re: bonferroni correction in multivariate regression

I haven't heard of paucity of effects but have heard of sparsity of effects which seems to be the same thing.

6. ## The Following User Says Thank You to Dason For This Useful Post:

rogojel (11-27-2014)

8. ## Re: bonferroni correction in multivariate regression

If we accept the idea for a moment that true hypotheses "b=0" exist,
then a singificant F test in mutiple regression suggests that at least
1 regression coefficient deviates from zero. If we have a model with
12 coefficients and 3 of them are "signficant", then 2 of them could
be false-positives (the actual probability that there are false-positives
depends on additional factors, but basically is > 0). So why don't we
have to adjust for multiple testing? Somewhere it was asked why we
have to adjust in ANOVA post hoc tests but not in case of regression
with dummy coding.

With kind regards

K.

9. ## The Following User Says Thank You to Karabiner For This Useful Post:

rogojel (11-30-2014)

10. ## Re: bonferroni correction in multivariate regression

Hi Karabiner,
exactly! That is what is bugging me too.

BTW I am just reading a book http://www.amazon.de/Data-Mining-Bus...words=Ledolter

and it has a whole chapter about this problem. The author basically suggest using the lasso regression to select parameters . The pb. is the lasso selected parameters might have a high p-value if the model is checked traditionally - I am not sure what to do then.

I already eliminated the non-significant terms to get a model with low p-values and this seemed to work, but I believe the other alternative is also accceptable, i.e to keep all the terms and screw the p-values.

regards

11. ## Re: bonferroni correction in multivariate regression

Think of what substantively a slope of 0 means. It means that empirically (and in practice this commonly means correlations not experimental test ) one variable has abolutely no relationship with another variable. It seems unlikely that a researcher would include many if any variables that are not somehow in the same domain in their analysis because placing totally non-sensical variables serves no research purpose and violates the concept of parsimony. Realistically IV and DV will be in the same dimension commonly even if the author feels one will not drive the other. So some correlation is always likely even if not very much.

Also by shear random chance it is likely that there might be some spurious correlation between variables(even if you had a population, but even more so if you have a sample). This might be pure noise, but it would likely occur. This ignores of course measurement error and movement over time which makes the problem worse.

So non-zero slopes are probably unrealistic - certainly in correlational studies which I imagine most regression addresses.

12. ## Re: bonferroni correction in multivariate regression

Originally Posted by rogojel
@injektilo - I do not see the practical difference. If I can not reject the null hypothesis that would mean, in practice,mthat I have no basis, for example, to request a new investment in a plant for some machine that will control that factor. If I can prove the null hypothesis false, I have the needed arguments to request such an investment.

regards
rogojel
The difference appears like splitting hairs, but it is nonetheless important. The difference is that you have to conclude that you don't have enough information to say the null hypothesis is false, rather than conclude by saying the null hypothesis is true. It's like saying that because you don't have enough evidence to say that something is not equal to 0, then it must be equal to 0. It's a leap in logic.

13. ## Re: bonferroni correction in multivariate regression

Originally Posted by Injektilo
The difference is that you have to conclude that you don't have enough information to say the null hypothesis is false
Yep. The logical argument:
I don't have enough information to reject the null hypothesis. That is, this test statistic would be quite probable if the null hypothesis was true (p > 0.05)
Therefore the null hypothesis is probably true.

Is obviously a fallacy. But what about the other possibility? I.e., What happens if the p value is statistically significant? Then you have this logical argument:

This test statistic would be improbable if the null hypothesis was true (p < 0.05)
Therefore the null hypothesis is probably false

But that's a logical fallacy too... (probabalistic modus tollens).

14. ## Re: bonferroni correction in multivariate regression

I don't know about baysian approaches, but I was always taught you could never conclude the null was true. It is either rejected or not rejected. That is why researchers set up the alternate hypothesis to test what they really believe is true. They are hoping to reject the null, because if they don't they have learned a lot less than if they do.

Not good statistics, but good if you want to get published since not finding something is a lot harder sell then finding something.

15. ## Re: bonferroni correction in multivariate regression

Originally Posted by noetsi
I don't know about baysian approaches, but I was always taught you could never conclude the null was true. It is either rejected or not rejected.
It depends a little on the framework. In Fisherian NHST, and the "hybrid" method most people learn now, you can't conclude the null is true.

But in Neyman-Pearson NHST, you can "accept" the null hypothesis if p > alpha. That said, one isn't strictly saying that the null hypothesis is true, just that you have evidence to justify acting as if it were true (i.e., you have grounds for a decision to guide behaviour).

In some Bayesian tests - e.g., some of the Bayes Factor tests being developed at the moment - it is possible to provide evidence to support a point null hypothesis. Which is useful if you're doing something like testing for extrasensory perception, where the exactly null hypothesis being tested might actually be true. But in Bayesian estimation more generally you typically specify a continuous prior probability distribution for the estimated parameters, which means you're implicitly saying that the probability that the null hypothesis is exactly true is zero.

That is why researchers set up the alternate hypothesis to test what they really believe is true.
In obscure tidbits of information for the day: Another way to use NHST is the "strong" form. I.e., You have a theory that makes a quantitative prediction about what the exact true value of the parameter should be (which might not be zero). You then specify this value as the null hypothesis, and see if you can find evidence to reject it. This use of NHST fits more with a Popperian approach to science (i.e, you specify a theory and then try to falsify it). I've never seen it done in practice - theories in psych are almost never specific enough to predict the exact value of a parameter. But apparently this approach has been used in physics.

16. ## Re: bonferroni correction in multivariate regression

In classes in social science it was drilled into my head over and over again you can only reject the null

In no research that I have been involved in, all social sciences, would you reasonably know an exact value for the null.

Page 2 of 2 First 1 2

 Tweet