+ Reply to Thread
Page 1 of 3 1 2 3 LastLast
Results 1 to 15 of 42

Thread: Practical significance

  1. #1
    Points: 1,589, Level: 22
    Level completed: 89%, Points required for next Level: 11

    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Practical significance




    The difference between two mean values is 103 units
    p-value is 0.00001 at alpha=0.05
    95% confidence interval is (50, 150)
    difference of 100 units is of practical importance.
    Would the mean difference be practically significant for the given difference between two mean values and confidence interval?

    Thanks!

  2. #2
    Points: 1,589, Level: 22
    Level completed: 89%, Points required for next Level: 11

    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Practical significance

    Quote Originally Posted by victorxstc View Post
    Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".
    i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.

  3. #3
    Pirate
    Points: 15,159, Level: 79
    Level completed: 62%, Points required for next Level: 191
    victorxstc's Avatar
    Posts
    875
    Thanks
    229
    Thanked 332 Times in 297 Posts

    Re: Practical significance

    Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".

    --------------

    Quote Originally Posted by _joey View Post
    i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.
    It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.

    In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.

  4. #4
    Points: 1,589, Level: 22
    Level completed: 89%, Points required for next Level: 11

    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Practical significance

    Quote Originally Posted by victorxstc View Post
    It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.
    But the difference calculated is _not_ 88 units, it's 103 units.

    In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
    The lower bound of CI should be above 100 or below -100 units?

  5. #5
    Pirate
    Points: 15,159, Level: 79
    Level completed: 62%, Points required for next Level: 191
    victorxstc's Avatar
    Posts
    875
    Thanks
    229
    Thanked 332 Times in 297 Posts

    Re: Practical significance

    even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.

  6. #6
    Points: 1,589, Level: 22
    Level completed: 89%, Points required for next Level: 11

    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Practical significance

    Quote Originally Posted by victorxstc View Post
    even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.
    I hear you. Some authority references would be nice to have to substantiate your comments because I haven't found anything on the web in regards to the CI bounds and practical significance.

  7. #7
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Practical significance

    To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  8. #8
    Points: 1,589, Level: 22
    Level completed: 89%, Points required for next Level: 11

    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Practical significance

    Quote Originally Posted by noetsi View Post
    To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
    That's what I thought too. Since, 100 units is of practical significance then the difference of 103 units would be practically significant too. victorxstc, however, is talking about CI bounds. I think he is referring to size effect.

  9. #9
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Practical significance

    I would argue, and the statisticians I have read, that elements such as CI bounds has nothing to do with substantive importance. Much of statistics gets at if the results you find could be tied to random error. That is entirely seperate from issue of whether the effect discovered (if real) matters.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  10. #10
    Devorador de queso
    Points: 95,922, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Practical significance

    So are you telling me that if the confidence interval had been (-1000, 1206) and the point estimate was 103 you would be completely fine saying that there is a practically significant difference?
    I don't have emotions and sometimes that makes me very sad.

  11. #11
    Pirate
    Points: 15,159, Level: 79
    Level completed: 62%, Points required for next Level: 191
    victorxstc's Avatar
    Posts
    875
    Thanks
    229
    Thanked 332 Times in 297 Posts

    Re: Practical significance

    @Joey

    It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more

    @noetsi

    I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).

  12. #12
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Practical significance

    If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.

    I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
    I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  13. #13
    Pirate
    Points: 15,159, Level: 79
    Level completed: 62%, Points required for next Level: 191
    victorxstc's Avatar
    Posts
    875
    Thanks
    229
    Thanked 332 Times in 297 Posts

    Re: Practical significance

    Dason, I mean both must be simultaneously be either above or below that margin. Your good example, not only crosses that margin, but also crosses zero so it is not only practically nonsignificant, it is even statistically nonsignificant

  14. #14
    Devorador de queso
    Points: 95,922, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Practical significance

    Yes but I was just trying to illustrate why it should matter that we take the confidence interval into account.
    I don't have emotions and sometimes that makes me very sad.

  15. #15
    Points: 1,589, Level: 22
    Level completed: 89%, Points required for next Level: 11

    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Practical significance


    Quote Originally Posted by victorxstc View Post
    @Joey

    It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more
    Can you recall what those sources were?

+ Reply to Thread
Page 1 of 3 1 2 3 LastLast

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats