Goodness of Fit Tests - are they really good?

[Dr.Pythagoras]

Hi,

after some time of using goodness of fit tests without really thinking about them, I came to this curious question:

Normally, it is recommended to have a Null hypothesis which you want to reject. (I.e. the opposite of what you want to show.) However, in Goodness of Fit this is not the case. And not being able to reject the Null hypothesis is in my understanding not really a strong case for the alternative hypotheses.

One might say that still the test is better than nothing. But what does it really get me if I can say that have no reason to reject the hypothesis that some data is Weibull distributed?

I guess this all boils down to the alpha vs. the beta error - is there a way to determine the probability (at least on a level of magnitude) for the latter?

Is there any mistake or omission in my reasoning?

 

[hlsmith]

There are lots of GoF tests. Typically showing that you can't reject the observed = predicted or that your data fit to a distribution. Some people have issues with GoFs, for example the Hosmer-Lemeshow test get some criticism.


So we are typically failing to reject that the data is not the same distribution. I am sure you can examine the power and alpha to see if those test are well powered. Though another issue is that with big datasets, some times it becomes difficult not to reject the null, but the observed distribution is close enough to make its use in analytics appropriate - if the assumption is robust. All in all, distribution interpretation can be pretty subjective in many scenarios given circumstances and endgame use.