Hypothesis-test: difference between paired means

#1
Dear All,
I have a doubt about the theory of matched pair t-test. I have the difference between two means and I wanna do an hyphothesis-test analysis. I have the difference-mean, the difference-standard deviation and the degree of freedom.

Let’s start that I wanna say that the difference is equal to 0. So my null-hypotheses says that the difference is equal to 0. The alternative hypothesis says no. It’s a two tail problem. The significance level is set to 0.05 (alfa).

So, happens that if P-value is greater than alfa then I cannot reject the null hypothesis. If P-value is less than alfa then I reject the null hypothesis. This is clear!

Now, I wanna say that the difference is greater equal than D. The alternative hypothesis says that the difference is less than D (One tail problem). The significance level is set again to 0.05.

So, happens like before. Looking at the P-value I can/cannot reject the null hypothesis.

My doubts comes out now. I have made some tests and I see that for less value of D I get rejected the null hypothesis. Increasing D, I’m increasing the P-value. The fact is that if I continue to increase D for very big values, I will always be able to NOT reject the null hyphotesis! Doesn’t exist a D value big enough to reject the null hypothesis!

I’m guessing that I use the one problem tail just to find the lower/upper limit where I can not reject the null hypothesis! Is this correct or I wrote something wrong in my explanation?

Thanks!
Solimyr
 

rogojel

TS Contributor
#2
hi,
generally you can not pick the null hypothesis - it is a given, more or less. So, in your case the null hypothesis will be that the difference is D or less and the alternative will be that the difference is larger.

I think this will solve your problem.

regards
rogojel
 
#3
hi,
generally you can not pick the null hypothesis - it is a given, more or less. So, in your case the null hypothesis will be that the difference is D or less and the alternative will be that the difference is larger.

I think this will solve your problem.

regards
rogojel
I'm still not so sure. Could you please explain what do you mean with "it is a given, more or less" ?

So who set the hypothesis need to know very well the problem and what could expect? But in this case I could set just the nulll hypothesis with an equal and not using greater than or larger than.

Could you please describe to me an example? Or some useful link. I've seen some website ()
and youtube lectures but I didn't come out to the answer of my question.
 

rogojel

TS Contributor
#4
hi,
a quick explanation of how tests work, maybe someone will make it more precise:

So, in all test a value is defined, the so called test statistic, and the distribution of this value computed for the case that the test assumptions and the null hypothesis are both true. This means, that the null hypothesis is part of the test definition in the same way the test statistic is - we cannot arbitrarily pick a null hypothesisnfor a test.

for example for a paired t-test the test statistic will be t-value calculated from the differences of the pairs. The assumption is that the differences are normally distributed and the null hypothesis is that there is no difference between the pairs, that is, the mean of the pairwise differences is zero.

Now, based on this assUmption it is relatively easy to define the distribution of the t-values. Then we calculate the t-value from our samples and based on the distribution we defined assuming the null hypothesis (that there is no difference between the pairs) we can calculate how probable it is to get the t-value we actually got from our sample IF the null hypothesis was true. That is called the p-value. If there is a sizeable probability to get the t value we got if there was in reality no difference between the pairs, we conclude that we have no reason to affirm that there IS a difference between the pairs as our result could be coming from absample where the null is true. This no proof that is avtually coming from such a sample, but based on our data we can not exclude this possibility.

If, on the contrary, we see that it would be very improbable to get the t value we got if the null hypothesis was true, then we can say that there is a strong indication, that the null hypothesis was in fact false.

The whole logic relies on the fact that we have the distribution of the test statistic when the null hypothesis was true, so we can not switch the null hypothesis or redefine it in any way, we habe to stick to the one defined with the test.

sorry for the length, I hope this males it a bit clearer.

regards
rogojel
 
#5
hi,
a quick explanation of how tests work, maybe someone will make it more precise:

So, in all test a value is defined, the so called test statistic, and the distribution of this value computed for the case that the test assumptions and the null hypothesis are both true. This means, that the null hypothesis is part of the test definition in the same way the test statistic is - we cannot arbitrarily pick a null hypothesisnfor a test.

for example for a paired t-test the test statistic will be t-value calculated from the differences of the pairs. The assumption is that the differences are normally distributed and the null hypothesis is that there is no difference between the pairs, that is, the mean of the pairwise differences is zero.

Now, based on this assUmption it is relatively easy to define the distribution of the t-values. Then we calculate the t-value from our samples and based on the distribution we defined assuming the null hypothesis (that there is no difference between the pairs) we can calculate how probable it is to get the t-value we actually got from our sample IF the null hypothesis was true. That is called the p-value. If there is a sizeable probability to get the t value we got if there was in reality no difference between the pairs, we conclude that we have no reason to affirm that there IS a difference between the pairs as our result could be coming from absample where the null is true. This no proof that is avtually coming from such a sample, but based on our data we can not exclude this possibility.

If, on the contrary, we see that it would be very improbable to get the t value we got if the null hypothesis was true, then we can say that there is a strong indication, that the null hypothesis was in fact false.

The whole logic relies on the fact that we have the distribution of the test statistic when the null hypothesis was true, so we can not switch the null hypothesis or redefine it in any way, we habe to stick to the one defined with the test.

sorry for the length, I hope this males it a bit clearer.

regards
rogojel
Thanks for all the answers!

Reading your post, seems to me that the null hypothesis in a difference pair must be zero and couldn't be differnet from zero. So we couldn't set another difference value in the null hypothesis that must be zero anyway. This seems to me wrongly because I think we could set the null hypothesis but maybe that's why I'm getting confused. Probably, I wrong to define the null hypothesis. The null hypothesis in a t-test must be zero but in a z-test could be different from zero.
Is it correct?

Thank you very much!!
 

rogojel

TS Contributor
#6
you are right, one can have a null of the form, the difference is less then a non-zero limit, and this is the reason why I said "more or less" in the first post. The point is though that the null hypothesis is always in the direction of the difference being LESS then a given limit. You can not have a null hypothesis of the type the difference is GREATER then a given limit for the paired t-test.

It is not IMO that such a test would be inherently impossible, but one would have to come up with a test statistic and a handy distribution of that statistic for the case where the difference is greater than a limit. The pairednt-test is just not such a test (and I do not know any that is, probably because it is a lot easier to calculate the distribution for the traditional null hypothesis.



regards
rogojel
 
Last edited:

CowboyBear

Super Moderator
#7
So, in all test a value is defined, the so called test statistic, and the distribution of this value computed for the case that the test assumptions and the null hypothesis are both true. This means, that the null hypothesis is part of the test definition in the same way the test statistic is - we cannot arbitrarily pick a null hypothesisnfor a test.
Hi rogojel,

I'm sorry but I'm going to have to disagree with this. It is entirely possible to pick null hypotheses other than zero. The "nil" hypothesis (a null hypothesis of a zero effect) is used very commonly, but it is by no means the only possibility. For a neat example, think about the Eddington study. This was a famous study of the differing predictions of Newtonian physics and general relativity with respect to how much light would bend when going around the sun. Newtonian physics implied that the the observed position of a star whose light was passing near the Sun should change by 0.87 arcseconds. General relativity implied that the difference would be 1.75 arcseconds. So a bunch of astronomers went out during an eclipse to test the difference.

In this scenario it might make sense to specify a null hypothesis of a difference of 0.87 arcseconds (the Newtonian prediction), and see if we could find evidence to reject Newton's theory. It might make sense to do the same for the relativistic prediction. But it would not make much sense to specify a null hypothesis of zero light deflection, since this would not allow us to work out which theory was better supported. (As it happens, I'm pretty sure the original authors did not actually use any significance tests, which is an interesting lesson in itself).

The specification of a null hypothesis is really up to the researcher. The idea is to specify a null hypothesis such that finding evidence to reject the null hypothesis would actually tell us something important about the world. Sadly, many researchers specify "nil" hypotheses even when a zero effect is entirely implausible to start with (especially in psychology, where everything correlates with everything). Rejecting a null hypothesis that we already know to be false is of pretty limited value.
 

Dason

Ambassador to the humans
#8
you are right, one can have a null of the form, the difference is less then a non-zero limit, and this is the reason why I said "more or less" in the first post. The point is though that the null hypothesis is always in the direction of the difference being LESS then a given limit. You can not have a null hypothesis of the type the difference is GREATER then a given limit for the paired t-test.

It is not IMO that such a test would be inherently impossible, but one would have to come up with a test statistic and a handy distribution of that statistic for the case where the difference is greater than a limit. The pairednt-test is just not such a test (and I do not know any that is, probably because it is a lot easier to calculate the distribution for the traditional null hypothesis.
Would you mind elaborating on this post? Because if I'm not misunderstanding what you're trying to say then I disagree with your post - so some clarification would be appreciated...
 

rogojel

TS Contributor
#9
Would you mind elaborating on this post? Because if I'm not misunderstanding what you're trying to say then I disagree with your post - so some clarification would be appreciated...

Hi Dason,
what I mean is that the hypothesis tests work in these steps:

1, Define a test statistic
2. Determine the distribution of the test statistic for the case of the null hypothesis
3' Calculate the probability of measuring a value of the test statistic as extreme as the one actually measured using the above distribution
4. Decide if this measurement could possibly have come from the distribution, if yes do not reject the null.

I think, that mathematically there is no reason, always to define a "no, or limited difference" type of distribution for a test, but the tests I am aware of are of this type, probably because it is much easier to calculate the distribution of the statistic for this case.

So, I believe that tests and null hypothesis are tied together and one has to use the null hypothesis that is defined for the test. E.g. it is not possible ito run a paired t-test with the null hypothesis that the difference is greater then a limit.

I am curious to see if I am wrong with this?

regards
rogojel
 

CowboyBear

Super Moderator
#10
it is not possible ito run a paired t-test with the null hypothesis that the difference is greater then a limit.

I am curious to see if I am wrong with this?
But a paired t-test is just a one-sample t-test on the differences. We can choose any null hypothesis we like for a one-sample t-test, and the test can be one- or two-tailed.
 

rogojel

TS Contributor
#11
Sorry I gotta go to work, will come back in the evening 6:30 AM here.

Quick reply though: the null hypothesos for the one-sample t is that the mean equals a given value IIRC, and not that it is greater than a value. The tails refer to the alternative hypothesis.
 
#12
Interesting! I dind't expect to open a discussion and make it hot! :D

I'm reading all your comments trying to understand your thinks related to the problems!

Thanks All!!!
 

Dason

Ambassador to the humans
#13
Sorry I gotta go to work, will come back in the evening 6:30 AM here.

Quick reply though: the null hypothesos for the one-sample t is that the mean equals a given value IIRC, and not that it is greater than a value. The tails refer to the alternative hypothesis.
Well you can actually think of it either way. But it's very possible to specify a null hypothesis that dictates a range of values for the parameter of interest (ie \(H_o: \mu \leq 0\) or \(H_o: \mu \geq 0\)).
 

rogojel

TS Contributor
#14
hi Dason,
this looks very interesting, I did a quick superficial search and found one weblink that actually lists the 3possible null hypothesis, here:

http://stattrek.com/hypothesis-test/paired-means.aspx

However, all the others and some books, The Handbook of Parametric and Non-Parametric statistical methods including, only discuss the case with the null being a constant value not an interval.

Also, in Minitab, which might not be a standard, the only way to set up the paired t-test is with by specifying a constant for the difference, not an interval.

I also checked the one sample t and z tests, and again, only this website is listing null hypothesis with intervals, all other sources are specifying the null with a constant.

So, what is going on?

ps. I am also a bit curious about how the distribution of the t or z value should look like if all we know is that the mean is, say, less then K.

Thanks for the great discussion!

regards
rogojel
 

Dason

Ambassador to the humans
#15
Well it doesn't really change much of anything if the test statistic falls in the region that is more likely under the alternative hypothesis. And if the test statistic doesn't fall in the region that is more likely under the alternative then you're sure not going to reject the null hypothesis anyways so it doesn't really matter anyways - so essentially nothing changes.
 

rogojel

TS Contributor
#16
Also on the site I found this analysis, and IMO this can not be right:

"State the hypotheses. The first step is to state the null hypothesis and an alternative hypothesis.

Null hypothesis: μ >= 110
Alternative hypothesis: μ < 110

Note that these hypotheses constitute a one-tailed test. The null hypothesis will be rejected if the sample mean is too small.

Formulate an analysis plan. For this analysis, the significance level is 0.01. The test method is a one-sample t-test.

Analyze sample data. Using sample data, we compute the standard error (SE), degrees of freedom (DF), and the t-score test statistic (t).

SE = s / sqrt(n) = 10 / sqrt(20) = 10/4.472 = 2.236
DF = n - 1 = 20 - 1 = 19
t = (x - μ) / SE = (108 - 110)/2.236 = -0.894

where s is the standard deviation of the sample, x is the sample mean, μ is the hypothesized population mean, and n is the sample size.

Since we have a one-tailed test, the P-value is the probability that the t-score having 19 degrees of freedom is less than -0.894."

When calculating the t-value the analysis uses the "the hypothesized population mean" but there is no such thing, the hypothesis is an interval. Also, I see no difference in the p-value calculation from the case where they usedd the null hypothesis mean = 110 and there has to be a difference, right?
 

Dason

Ambassador to the humans
#17
I don't really want to get into the whole theory of it all right now but it's just a likelihood ratio test: http://en.wikipedia.org/wiki/Likelihood-ratio_test

You would use the "most likely" value of the parameter under the null hypothesis when calculating your p-value. If the sample mean in your case was above 110 then it boils down to the exact same thing you would do if you specified your null as \(\mu = 110\).
 

rogojel

TS Contributor
#18
Well it doesn't really change much of anything if the test statistic falls in the region that is more likely under the alternative hypothesis.
i would still see two problems:
1. how to define a t-value in this case?( x-mu)/s will not work because we have no defined mu.
2. how to calculate the probability of the statistic when we have an infinite number of possibilities (one for each possible value of mu) for the mean, each of those contributing a small amount to the probabilty?

regards
rogojel
 

rogojel

TS Contributor
#19
hi,
that means that from the calculations POV it actually does not matter whether one specifies a constant as the null or a semi-infinite interval? this is really counter-intuitive, I would definitely expect different p-values.

From a theoretical perspective there is a huge difference IMHO: Imagine I wanted to sell homeopathic medicine and set up a test to prove that my medicine works. I set up zhe test with the null being - my effect is greater or equal to zero. Then I ran the test, with the exact same calculations as if the null was : the effect is zero. I then find that I can not reject the null - under the constant null this means that I can not make the claim that my treatment works. Under the interval null the exactly same test means that my opponents can not make the claim that it does NOT work.

what do you think?
 

rogojel

TS Contributor
#20
I don't really want to get into the whole theory of it all right now but it's just a likelihood ratio test: http://en.wikipedia.org/wiki/Likelihood-ratio_test

You would use the "most likely" value of the parameter under the null hypothesis when calculating your p-value. If the sample mean in your case was above 110 then it boils down to the exact same thing you would do if you specified your null as \(\mu = 110\).
hi Dason,
just a last thought. I do not know the theory, but just from a practical point of view, how can the most likely value be the lower edge of the interval? I make the hypothesis that the mean is greater or equal to a value D. What assumptions do I need to make to have the most likely value be equal to D?