The difference between two mean values is 103 units
p-value is 0.00001 at alpha=0.05
95% confidence interval is (50, 150)
difference of 100 units is of practical importance.
Would the mean difference be practically significant for the given difference between two mean values and confidence interval?
Thanks!
Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".
--------------
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.
In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
But the difference calculated is _not_ 88 units, it's 103 units.
The lower bound of CI should be above 100 or below -100 units?In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.
To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
"Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995
I would argue, and the statisticians I have read, that elements such as CI bounds has nothing to do with substantive importance. Much of statistics gets at if the results you find could be tied to random error. That is entirely seperate from issue of whether the effect discovered (if real) matters.
"Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995
So are you telling me that if the confidence interval had been (-1000, 1206) and the point estimate was 103 you would be completely fine saying that there is a practically significant difference?
I don't have emotions and sometimes that makes me very sad.
@Joey
It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more
@noetsi
I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.
I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
"Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995
Dason, I mean both must be simultaneously be either above or below that margin. Your good example, not only crosses that margin, but also crosses zero so it is not only practically nonsignificant, it is even statistically nonsignificant
Yes but I was just trying to illustrate why it should matter that we take the confidence interval into account.
I don't have emotions and sometimes that makes me very sad.
Tweet |