Few Questions about P-Value

[anoopbal]

This is my first post and I had a few questions which I feel I need help. I have read a few textbooks and papers but I am not sure hence.

1. From what i understand any study results can be made statistically significant with a large enough sample size? Is that true?

2. If a study shows that Group A mean is 5% greater than Group B, does it mean if i have bigger sample size Group A mean will only get bigger? Or it can also go in the reverse direction (where Group B mean gets bigger)?

Thanks for the help

 

[Dason]

1. From what i understand any study results can be made statistically significant with a large enough sample size? Is that true?

Yes and no. Ultimately it's almost never true that the null hypothesis is true. If you think about it there will almost always be some sort of difference in means/variance/whatever_you're_looking_at between the groups. So if you increase the sample size you increase the power to detect that difference. So yes, with a large enough sample you will most likely reject the null. Now that doesn't mean the difference is practically significant though. Also, if by some chance the null hypothesis really is true then by increasing sample size you keep the same type 1 error rate.

2. If a study shows that Group A mean is 5% greater than Group B, does it mean if i have bigger sample size Group A mean will only get bigger? Or it can also go in the reverse direction (where Group B mean gets bigger)?

No, it could get a little smaller, it could get a little bigger. (There's more to it than this but oh well...) But the effect size isn't magically going to get larger just because you take a larger sample size. That doesn't make sense. The result might become more significant (smaller p-value) because you have a better estimate of the error and the standard error of the difference will decrease with a larger sample size. But that doesn't mean the estimate effect itself will get larger.

Ultimately with a larger sample size you're getting a better estimate of what the true means actually are.

 

[anoopbal]

Thanks a lot for the reply. Makes more sense now. I hope it is ok to ask a few more questions:

1. Could we say that the same size represents the population. Or is there any equation to find out if the sample size which represents the population the best. I have come across sample size calculation for descriptive studies, but what about hypothesis studies.

Or generalization is all about random sampling and random assignment and has very little do with sample size.

I have come across a number 30. If a group has 30 or 20 subjects it is said to represent the population. Is this true?

 

[anoopbal]

So for example, if the mean of group A is 20 and mean of group B is 10. The means are not statistically significant here. But if we had more sample size, the difference would be statistically significant. But the difference can also happen to be in favour of the group B. That is Group B could be 20 and Group A could be 10.

So statistical significance do not give you the direction, and the direction can go either way with more sample size. Does this make any sense?