Regarding Cohen's d use...

#1
Hi,

I am confused regarding added value of Cohen's d. Consider a case of paired t-test. I calculate the Cohen's d as:
mean(vec1) - mean(vec2)) / population_std;
population_std: standard deviation of x – y.

1. I am said that p-value is not informative enough because effect size might be tiny, but p-value is huge (e.g.: http://blog.stata.com/tag/cohens-d/). If we take two vectors of length 10 , the one contains '1' values and the second the 1.01 and 1.02. Running paired t-test will give very significant p-value, but the effect size is also huge (close to 2). However, the difference between 1.01 and 1 is tiny. Obviously, what plays a role here is a small std. But std is also part of p-value calculation.

2. When I run a simulation for randomly generated two vectors. If the length of the vectors is kept constant, the correlation between cohen's d and tstat is 1. When the vector length varies for each round, the correlation is R=0.95, which is also close to perfect.

3. In a my real case example, I had two types of measurements (for both before and after). In the first, I had 13 samples and I got t-stat=2.41 and cohen's d=0.67. In the second, I had 8 samples, and I got t-stat=2 and cohen's d=0.71. So, for larger sample size the stat value tends to be more significant while the effect size is weaker?

But all these differences are very marginal, so does cohen's d has too much added value?

Thank a lot,
John
 
#2
So, for larger sample size the stat value tends to be more significant while the effect size is weaker?
No - for larger sample size the p-value tends to be more significant, but effect size is independent of sample size (there is no 'n' in the formula for Cohen's d).

And that's the only real value of Cohen's d, it measures the size of the treatment effect in standard deviation units, rather than in standard error units. Thus its magnitude isn't influenced by n.
 
#3
Great, thanks a lot!
It makes sense. In my simulation I had too small variation of sample size. Now, I varied it from 1 to 10.000, and small sample sizes are the clear outliers in the t-stat vs. cohen's d correlation.