1. ## Practical significance

The difference between two mean values is 103 units
p-value is 0.00001 at alpha=0.05
95% confidence interval is (50, 150)
difference of 100 units is of practical importance.
Would the mean difference be practically significant for the given difference between two mean values and confidence interval?

Thanks!

2. ## Re: Practical significance

Originally Posted by victorxstc
Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".
i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.

3. ## Re: Practical significance

Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".

--------------

Originally Posted by _joey
i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.

In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.

4. ## Re: Practical significance

Originally Posted by victorxstc
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.
But the difference calculated is _not_ 88 units, it's 103 units.

In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
The lower bound of CI should be above 100 or below -100 units?

5. ## Re: Practical significance

even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.

6. ## Re: Practical significance

Originally Posted by victorxstc
even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.
I hear you. Some authority references would be nice to have to substantiate your comments because I haven't found anything on the web in regards to the CI bounds and practical significance.

7. ## Re: Practical significance

To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.

8. ## Re: Practical significance

Originally Posted by noetsi
To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
That's what I thought too. Since, 100 units is of practical significance then the difference of 103 units would be practically significant too. victorxstc, however, is talking about CI bounds. I think he is referring to size effect.

9. ## Re: Practical significance

I would argue, and the statisticians I have read, that elements such as CI bounds has nothing to do with substantive importance. Much of statistics gets at if the results you find could be tied to random error. That is entirely seperate from issue of whether the effect discovered (if real) matters.

10. ## Re: Practical significance

So are you telling me that if the confidence interval had been (-1000, 1206) and the point estimate was 103 you would be completely fine saying that there is a practically significant difference?

11. ## Re: Practical significance

@Joey

It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more

@noetsi

I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).

12. ## Re: Practical significance

If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.

I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.

13. ## Re: Practical significance

Dason, I mean both must be simultaneously be either above or below that margin. Your good example, not only crosses that margin, but also crosses zero so it is not only practically nonsignificant, it is even statistically nonsignificant

14. ## Re: Practical significance

Yes but I was just trying to illustrate why it should matter that we take the confidence interval into account.

15. ## Re: Practical significance

Originally Posted by victorxstc
@Joey

It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more
Can you recall what those sources were?

Page 1 of 3 1 2 3 Last

 Tweet