t-e test p value interpretation

#1
Hi All

*N.b. title should read t-test*

1st post, I need some help interpreting the results of a t-test. I understand that t-tests normally only reject the null hypothesis if p<0.05. This is commonly described as being "statistically significant". However, I am interested in what statements can be made if the p value is higher than 0.05.

If I do a t-test and find that the p value is 0.02 can I say that the probability of the data I have tested (e.g. comparing two mean values derived from two groups of scores on exam results) being due to chance is 2%?

Can I then say that the probability of it being due to factors other than chance is 98%?

And to continue this logic, can I then say that a p value of 0.30 indicates that there is a 30% chance that the difference observed is due to chance, and hence there is a 70% chance that it is due to factors other than chance?

Can I design a test with the parameter set at 0.50? If I get a P value of 0.4 can I then accept my null hypothesis and state that there is a 60% chance that the data is due to factors other than chance?

I know statisticians have a love affair with 95% confidence intervals, but surely the p value tells us something below this level, even if it offers a lot less certainty than at 95%?

Thanks, all views appreciated!
 
Last edited:
#2
Soooo.... You asked for a language check. I will do my best, but I warn you that I am notoriously bad at plain english answers. So I apologize if anything I say is wrong =)~ Its possible.



I understand that t-tests normally only reject the null hypothesis if p<0.05.

I would toss out the word normally and replace it with commonly. No particular alpha is preferred enough to call it normal.

This is commonly described as being "statistically significant".

More precisely it is statistically significant evidence that the null hypothesis is not true. The p-value itself can be defined outside of a probability framework as the lowest significance level for which the null hypothesis would be rejected.


If I do a t-test and find that the p value is 0.02 can I say that the probability of the data I have tested (e.g. comparing two mean values derived from two groups of scores on exam results) being due to chance is 2%?

Close but not quite. The probability is of getting that result or one more extreme under the null hypothesis (and associated assumptions about the modeling). Key here is in a continuous distribution the probability of observing anything specific is zero. So we need a range of values to make a greater than zero probability.

In this particular case the probability of observing the t statistic that we did or any other t statistic greater in magnitude.

It is worth noting though how this range of more extreme values is detirmined is a topic of debate in more general settings that can change everything (for example 2 by 2 tables).

Now there is one other thing you put in there that is wrong. You wrote: "probability of [] being due to chance is 2%?"

No this or something more extreme would happen 2% of the time under chance. But the probability of this event happening being due to chance is undefined. Parse that carefully. In your way we assume a model, a probability structure, and we say what is the probability that we observe... "this". In yours you tell us about the probability the entire thing is chance which is much different.

Can I then say that the probability of it being due to factors other than chance is 98%?

You cannot. Here is a crucial point that instructors love to ping people on... it boils down to this: reality is reality with probability 1. The probability that it is due to factors other than chance is 1 or 0. It is true or false. Either the null hypothesis was true and the result was due to chance or it was not. 1 or 0. Though you can see notation abused in this area. I've seen Rice murder notation in this regard.

I think the uncomfortableness comes from people longing for stats to tell us the answers. So if I can tell you the probability the null hypothesis is not true that is comfortable. But Neyman-Pearson (what you use) statistical inference cannot due that. It can only tell you the probability of observing this or something more extreme under the null assumptions. Which is far and away from the probability that the null assumptions are true imo ( I am starting to stretch into original thought).



And to continue this logic, can I then say that a p value of 0.30 indicates that there is a 30% chance that the difference observed is due to chance, and hence there is a 70% chance that it is due to factors other than chance?

No again. Its the probability if it were due to chance not the probability that it is due to chance. (hey I think I said something clear there. That's a first for me)

If I get a P value of 0.4 can I then accept my null hypothesis

There is a lot of people that would say "you can never accept a null hypothesis". So you probably shouldn't. And when you think about it what evidence do you have really that the null is true?

Would you accept this: X implies Y. Proof: I cannot disprove it therefore it is true. Don't you want something more? Don't you want some sense of what ideas have been exhausted before you believe it?

It is possible the inability to reject the null will be constructed in a stronger framework as a good reason to believe it is true. But in general the normal Neyman-Pearson thought process is very weak here so it should be avoided.
 
#3
So alot of my answer boils down to this: the probability under the assumption of chance is not the same as the probability of chance at work. And I am a little worried I am wrong [at the time of writing this sentence but not at the end.]

You know now I think about it I have heard some instructors butcher this, and I don't think they were correct.

Hmm brainstorming:

So lets think carefully about type I error. This is the probability of rejecting a null hypothesis that is true. The alpha is the type I error rate. This fits with what I am saying because here the entire aparatus associated with the null is assumed.

Type II error . This is the probability of failing to reject even though the alternative is true. So key here is there are quite a bit of assumptions that come with the alternative that are commonly ignored. For example means of a normal distribution not being equal to a specific value We can calculate type II error given the assumption of normality, but without that assumption we cannot.

Alright I am comfortable in my answer:
In general the probability under the assumption of chance is not the probability of being due to chance.

Avoid thinking it is and you will have a stronger thought process.


On a side note when you wrote:
If I get a P value of 0.4 can I then accept my null hypothesis

the key phrase that is often used in the thought process here is "insufficient evidence to reject the null hypothesis". Very handy thought/phrase in statistics. I use it a lot.