Application Specific: Strange behaviour of students T-test compared to std.dev plots

ngio

New Member
#1
Hello all,
********************************************************************
SEE THE FITH POST OF THIS THREAD TO GET A BETTER IDEA OF THE ISSUE
********************************************************************
I am comparing three datasets of length 7201 datapoints each (on images this is the x-axis values). At every one of the 7201 datapoints I have 989, 1003, 945 trials each (on the figures this is the z-axis values). think of it as a matrix, 7201 x ~1000 for each dataset. I take the means and standard deviations in the '1000' direction to provide a 7201 x 1 array for the mean (then do the same for the Std.Dev). Now when I plot this data they overlap according to the following figure (data-means.jpg), the top hump is the mean+1\sigma, the bottom hump is the mean-1\sigma, and the middle hump is the mean of each dataset.

So graphically I can conclude that the three datasets are the same, however when I conduct the two tailed t-test (as well as the F-test of variance) comparing each location within the 7201 array i get the following probability plots.

probability 1-2.jpg is the p-value computed from the t-test when comparing data1 to data2.
probability 1-3.jpg is the p-value computed from the t-test when comparing data1 to data3.
probability 2-3.jpg is the p-value computed from the t-test when comparing data2 to data3.

The result of the t-test shows that they are "at times" statistically the same however at other times not equivalent. I can understand this for (x-values) of -360 to ~0 although from the (x-values) 0 to 100 they should be statistically the same according to the standard deviation plots. Is this conclusion not what I should expect?

Perhaps someone with more experience in t-test may be able to help determine which conclusions I can draw from this data. To me it simply doesn't make any sense.
 
Last edited:

Dason

Ambassador to the humans
#2
Re: Application Specific: Strange behaviour of students T-test compared to std.dev pl

I don't understand what you're saying and you didn't really explain what your plots are visual representations of. Please provide more details. If all you're asking if it's possible for the data to visually look similar but for a t-test to show significant differences then yes this is very possible and it might even be expected with the sample sizes that you have.
 

ngio

New Member
#3
Re: Application Specific: Strange behaviour of students T-test compared to std.dev pl

My apologies, I tried to keep the post short. I will re-write it more explicitly and with more detail.
 

Dason

Ambassador to the humans
#4
Re: Application Specific: Strange behaviour of students T-test compared to std.dev pl

There are just weird things in your posts. Like you say "probability 1-2.jpg is the p-value computed from the t-test when comparing data1 to data2" but I don't understand how that plot is representing a p-value.
 

ngio

New Member
#5
Re: Application Specific: Strange behaviour of students T-test compared to std.dev pl

Let's try this again... This time I will take the approach of using the simplest form of my issue.


I have three datasets,

data1 = 967 datapoints,
data2 = 981 datapoints,
data3 = 963 datapoints.

If I plot the mean and their standard deviations (i.e. mean+/-1std.dev) I produce figure 'means.jpg'.

When I plot their Histograms (figure 'Histograms.jpg') you can see that they overlap heavily.

However when i conduct the t-test between each of the datasets I get the following results.

Data1 Vs. Data2:
H = accept, p = 495.84e-24

Data1 Vs. Data3:
H = reject, p = 0.98552

Data2 Vs. Data3:
H = accept, p = 1.3788e-21

It simply does not make sense to me. From what I understand, the t-test is saying that the Data1 and Data2 are significantly different as well as comparing Data2 and Data3 they are significantly different. This is seen by the extremely low p-values. However comparing Data1 to Data3, it claims that they are not significantly different.

Your comment about the sample size is interesting, can this be the root cause to my conundrum?

hopefully this was a little more clear.
 
Last edited:

BGM

TS Contributor
#7
Re: Application Specific: Strange behaviour of students T-test compared to std.dev pl

From your error-bars plot, it looks like the sample mean of the data 2 is lower than those in data 1 and data 3 by 0.4 SD of the sample. Note that the SD of the sample mean actually is decreasing with the sample size (divided by \( \sqrt{n} \) from the SD of the population) - the variation will become less, and you estimate will become more accurate and reliable. Therefore with your sample size, actually when you are comparing them and try to obtain a p-value, it is just like asking "what is the chance to obtain a point from a normal distribution which is away from mean by at least \( 0.4 \times \sqrt{n} \) SD, or approximately 8 SD".

It is not surprising to obtain such a significant p-value.

P.S. the number 0.4 is just a very rough visual estimate.
 

ngio

New Member
#8
Re: Application Specific: Strange behaviour of students T-test compared to std.dev pl

Hi BGM,

Firstly, thank you for taking the time to look into my issue, its greatly appreciated.

what you said makes sense. Pretty sure that clears things up for me.

Thanks again for your help.

Regards,
ngio