Have I interpreted these results correctly? Please Help!

[Chloe Randles]

I am a masters student critically analysing meta‐analysis, please help

The study found a small significant effect size using a Cohen’s d test d = 0.28, 95% CI, p < 0.001. Is this a correct interpretation?

The result of the odd ratio test was significant, OR = 1.79, 95% CI, p  < 0.001. Is this correctly written?

However, when analysing the effect of the test, significant heterogeneity in the outcome effect sizes (d’s) between studies was found, Q(50) = 657.3, p < 0.001, I2 = 92.39. This static is extremely high and indicates that 92.39% of the results were due to heterogeneity and shows inconsistency across the studies used. Have I written this correctly?


Contrastingly, on the other effect test significant heterogeneity in the study was not found, Q(27) = 34.76, p = 0.15, I2 = 22.33, which suggests the effect found was not due to heterogeneity. Is this correctly interpreted?

A meta-regression was conducted to investigate the heterogeneity, the preference outcome effect was found to depend on the study design, QB(3) = 13.76, p = 0.003, the type of outcome that was assessed, QB(2) = 13.60, p = 0.001, and the timing of outcome measurement, QB(1) = 5.73, p = 0.02. Is this written as it should be?

The overall effect in this meta-analysis was small, d = 0.28, but significant. Is this correct?


Any help would be amazing!!! I'm a counselling student so statistics isn't really my thing and I'm really struggling!!

 

[Buckeye]

If you took these statistics straight from a paper and you can site the paper then there is probably no issue with the way you presented them. But, you are asking if you correctly interpreted some results. We have no context of the analysis, so it's difficult to assess that part. But again, since you are siting results from a paper, I would think that the paper explains itself.

 

[Chloe Randles]

If you took these statistics straight from a paper and you can site the paper then there is probably no issue with the way you presented them. But, you are asking if you correctly interpreted some results. We have no context of the analysis, so it's difficult to assess that part. But again, since you are siting results from a paper, I would think that the paper explains itself.

Hey, thanks for a fast response!!!

The main bit I'm not sure about is this bit 'However, when analysing the effect of the test, significant heterogeneity in the outcome effect sizes (d’s) between studies was found, Q(50) = 657.3, p < 0.001, I2 = 92.39. This static is extremely high and indicates that 92.39% of the results were due to heterogeneity and shows inconsistency across the studies used.' does that statistic show that that 92.39% of the results were due to heterogeneity and shows inconsistency across the studies used?

Also are these results from a meta regression ? QB(3) = 13.76, p = 0.003, the type of outcome that was assessed, QB(2) = 13.60, p = 0.001, and the timing of outcome measurement, QB(1) = 5.73, p = 0.02.

I know I'm probably not making much sense but if you can help I'd be so grateful!

 

[Buckeye]

I think you are okay. Basically, what the first piece is saying is that the effect sizes amongst the studies are not the same. In my experience, "psychological" research is difficult to reproduce because the outcome variables are often latent variables and they cannot be measured easily. What outcome are they studying? I'm making a big assumption that they are measuring some sort of mood/attitude variable.

 

[Chloe Randles]

I think you are okay. Basically, what the first piece is saying is that the effect sizes amongst the studies are not the same. In my experience, "psychological" research is difficult to reproduce because the outcome variables are often latent variables and they cannot be measured easily. What outcome are they studying? I'm making a big assumption that they are measuring some sort of mood/attitude variable.

Thank you!! As long as the heterogeneity bit makes sense. It is a meta-analysis about how matching clients preferences increases outcomes and lowers therapy dropout rates

 

[hlsmith]

Well, they can have a bunch of study heterogeneity, that isn't necessarily an issue. It is good to report this and then the investigator should move forward using random effects models to account for the random error (due to sampling bias) and study heterogeneity (studies may be examining different super populations or things). I^2 is the proportion of variability accounted for via study heterogeneity. For example you may study patients at a community hospital and I study patients on a large academic hospital. And your community hospital may be a part of a health system while someone else's is not. Heterogeneity, but you just need to control for hospitals AKA studies in model. Doesn't negate results.

Side comment, we also don't know how many studies were used in the MA? Much like multilevel models, only having a couple of clusters (e.g., few studies in MA) can be problematic especially if their is heterogeneity. MA stats aren't that hard. Refer to Borenstein, Hedges, Higgins, Rothstein's book Introduction to Meta-Analysis, which is typically free online.

 

[Chloe Randles]

Well, they can have a bunch of study heterogeneity, that isn't necessarily an issue. It is good to report this and then the investigator should move forward using random effects models to account for the random error (due to sampling bias) and study heterogeneity (studies may be examining different super populations or things). I^2 is the proportion of variability accounted for via study heterogeneity. For example you may study patients at a community hospital and I study patients on a large academic hospital. And your community hospital may be a part of a health system while someone else's is not. Heterogeneity, but you just need to control for hospitals AKA studies in model. Doesn't negate results.

Side comment, we also don't know how many studies were used in the MA? Much like multilevel models, only having a couple of clusters (e.g., few studies in MA) can be problematic especially if their is heterogeneity. MA stats aren't that hard. Refer to Borenstein, Hedges, Higgins, Rothstein's book Introduction to Meta-Analysis, which is typically free online.

Thank you, so the researcher did the Q tests after, so was this to study heterogeneity? Also there were 28 studies with over 16000 participants in today.

 

[Chloe Randles]

Thank you, so the researcher did the Q tests after, so was this to study heterogeneity? Also there were 28 studies with over 16000 participants in today.

This section is assessing the heterogeneity isn’t it … meta-regression was conducted to investigate the heterogeneity, the preference outcome effect was found to depend on the study design, QB(3) = 13.76, p = 0.003, the type of outcome that was assessed, QB(2) = 13.60, p = 0.001, and the timing of outcome measurement, QB(1) = 5.73, p = 0.02?