Quick Question on Hypothesis Testing

[icecream12345]

Let's say you test hypothesis at alpha=.05, does that mean that 5% of your rejections will be wrong?

Is this statement true? If not, can you give an example?

I know that alpha= P(type 1 error), which is the probability of rejecting given Null is true. But I still can't tell whether that statement is true or not. Some of my friends are saying that it is false, but I am not totally convinced.

Am I missing something? Can someone please help me understand this?

 

[Karabiner]

Let's say you test hypothesis at alpha=.05, does that mean that 5% of your rejections will be wrong?

If you only test true null hypotheses, then 5% of your rejections will be wrong.
If you test only false null hypotheses, 0% of your rejections will be wrong.
If you test a mixture of wrong and false null hypotheses, then between 0% and 5% of your rejections will be wrong.


With kind regards

K.

 

[icecream12345]

Thanks! I was just reaching to this conclusion and really wanted to hear from somebody else too.

 

[Karabiner]

If you only test true null hypotheses, then 5% of your rejections will be wrong.
If you test only false null hypotheses, 0% of your rejections will be wrong.
If you test a mixture of wrong and false null hypotheses, then between 0% and 5% of your rejections will be wrong.


With kind regards

K.

Sorry, but that was misleading, as I see now.

If you test only true Null Hypotheses, then of course
all of your rejections must be wrong. But you can expect
that you will make wrong (i.e. false-positive) decisions
in 5% of tests (while false-negative decisions are impossible
if all Null Hypotheses were true).

 

[hlsmith]

Karabiner, your post #4 is confusing. At least the latter part.

 

[Karabiner]

If all Null Hypotheses are true, then it is impossible to make a type II error.
Therefore, the expected proportion of false decisions is 5%.

(at least, if I didn't confuse things once more)