Composite reliability vs Cronbach's alpha

#1
Hello

Wondering if someone could clarify

I'm developing a new measure with 10 constructs. Each construct has between 4 - 6 items.

I've been assessing the reliability for each construct using both Cronbach's alpha and composite reliability scores.

For most constructs, both reliability scores are similar. However for some constructs, Cronbach's alpha is higher than the composite reliability score (e.g. 0.91 vs 0.85 respectively).

This is strange to me, as my understanding is that Cronbach's alpha is a lower bound estimate of composite reliability, thus would generally be lower.

Wondering if people have come across this?

Many thanks in advance
 

hlsmith

Not a robit
#2
Not familiar with the term "composite reliability score". Can you describe what procedure you are using in particular?
 

spunky

Doesn't actually exist
#3
This is strange to me, as my understanding is that Cronbach's alpha is a lower bound estimate of composite reliability, thus would generally be lower.
alpha is a lower bound to reliability (in the population) IF AND ONLY IF the model in the population is a one-factor model... and it would still only be lower in the population, not necessarily in the sample.

the fact that you have 10 constructs is already telling you that the calculation of alpha is somewhat meaningless. besides, if you have error covariances or crossloadings (which is highly likely given the complexity of the SEM model) then alpha can be lower, higher or equal than the composite reliability. the lower bound definition really only holds under very contrived circumstances.
 
#4
Many thanks Spunky (good username)

So just to clarify - it would be best to report composite reliability scores.

Much appreciated ! =)




alpha is a lower bound to reliability (in the population) IF AND ONLY IF the model in the population is a one-factor model... and it would still only be lower in the population, not necessarily in the sample.

the fact that you have 10 constructs is already telling you that the calculation of alpha is somewhat meaningless. besides, if you have error covariances or crossloadings (which is highly likely given the complexity of the SEM model) then alpha can be lower, higher or equal than the composite reliability. the lower bound definition really only holds under very contrived circumstances.
 

spunky

Doesn't actually exist
#5
So just to clarify - it would be best to report composite reliability scores.
YES! always. cronbach's alpha usually get reported mostly because it's tradition and because it pisses off certain people. so the scientist in me tells me "if you have access to composite reliability scores, report those" but the child in me says "always report alpha so people can look at it and lose their #@$%@!"
 

spunky

Doesn't actually exist
#8
I'm guessing "composite reliability" here is synonymous with coefficient omega?

omega... rho.... composite reliability... it goes by many names, with each researcher lending its name (i think it's McDonald's omega, Bentler's rho and Raykov's composite reliability)
 

Jake

Cookie Scientist
#9
Okay, well one thing that still doesn't make sense to me is: if the OP's model has no cross-loadings and no correlated errors, so that it is basically just 10 separate congeneric scales, then shouldn't it still be the case that alpha <= omega for each construct? I understand that under realistic assumptions alpha might under- or over-estimate reliability, but the model in question is devoid of any of those realistic assumptions, right? What am I missing?

anon2000, what software are you using?
 

spunky

Doesn't actually exist
#10
Okay, well one thing that still doesn't make sense to me is: if the OP's model has no cross-loadings and no correlated errors, so that it is basically just 10 separate congeneric scales, then shouldn't it still be the case that alpha <= omega for each construct? I understand that under realistic assumptions alpha might under- or over-estimate reliability, but the model in question is devoid of any of those realistic assumptions, right? What am I missing?
my assumption would be that there are no lower bounds for alpha for finite samples, as elaborated in Raykov (1997). for finite samples, i've seen alpha before be slightly higher than composite reliability (because really the issue here is a difference of barely 0.06). usually as the samples grow larger (and if the model is correct) alpha tends to go back to its usual role as a lower bound.
 

Jake

Cookie Scientist
#11
Interesting, I didn't realize that the lower bound thing only held in the limit with population values.

One other thing that occurred to me is that the OP's model probably has intercorrelations among the latent factors. I'm not sure how the software used by the OP is calculating what it calls simply "composite reliability" but is it possible these correlations are being taken into account? For example, one possibility would be that in the case of correlated factors, the "true score variance" part of the reliability equation would use only the unique true score variance, so that having correlations with the other factors would lower reliability since unique true score variance <= total true score variance. I'm not sure if people actually do this when estimating reliability though, or if it's instead conventional to ignore the factor correlations so that "composite reliability" really is equivalent to doing omega separately for each scale. Do you know?
 

CB

Super Moderator
#14
I didn't think that Raykov's composite reliability was the same as coefficient omega. But maybe they're connected in a way that I didn't know?

Like spunky says the whole idea of alpha being a lower bound on reliability is just not true. I see it repeated in undergrad methods texts that wouldn't touch concepts like tau equivalence and correlated errors, and yet somehow feel that it's still ok to assure students that alpha is "conservative". It isn't, or at least, not always. When there is positively correlated error across items, alpha can overestimate reliability (where reliability is defined in the CTT sense as true score variance / total variance).

Raykov's composite reliability coefficient allows you to take into account correlated error, and when this is done I've found that it usually provides a lower estimate of reliability than alpha does (if your model includes any correlated error terms). Any downward bias caused by a breach of tau-equivalence seems to get rapidly overwhelmed by upward bias from correlated errors. (anecdotal evidence alert here).

OP, how did you calculate the composite reliability?

Reference with formula for Raykov's coefficient for anyone interested:
Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35(2), 299–331. doi:10.1016/S0005-7894(04)80041-8
 

Lazar

Phineas Packard
#15
I cannot remember which paper it was but there was a peer reviewed article that described discussions about what is the true glb as dressing up in a suit to wrestle a pig. Anyone remember which paper that was?
 
#16
Hello - interesting conversation, and shows there is alot to learn about reliability calculations !

In response to your questions - yes, I've been using Raykov's composite reliability approach in Mplus to calculate composite reliability scores...
 

spunky

Doesn't actually exist
#17
ok, so a few comments (while this is still relevant, lol) that i couldn't bring forth before because they were gonna take some time.

I didn't think that Raykov's composite reliability was the same as coefficient omega. But maybe they're connected in a way that I didn't know?
lemme quote from one of the most comprehensive bibles of SEM out there: the EQS Manual (we're going old school, yaaay!). on page 119 where it talks about reliability it says:

Cronbach’s alpha (1951) is well-known. Developed for EQS, rho is based on the latent variable model being run. This could be any type of model with additive error variances. When this is a factor model, this gives Bentler’s (1968, eq.12) or, equivalently, Heise and Bohrnsted’s (1970, eq. 32) omega for an equally-weighted composite. With the /RELIABILITY command, it is based on a one-factor model (Raykov, 1997ab), and then rho is the same as McDonald’s (1999, eq. 6.20b) coefficient omega. Note that McDonald’s omega is a special case of Heise and Bohrnsted’s omega.

this is why i said, for practical purposes, they're all sorta doing the same thing. particularly beacuse reliability is usually thought about within the (hopefully), one-factor, tau-equivalent model.

Interesting, I didn't realize that the lower bound thing only held in the limit with population values.
Jake, let me show you a quick simulation i cooked up in lavaan to sort of make my point:

Code:
library(lavaan)
set.seed(123)

model <- '
	   f1 =~ .7*V1 + .7*V2 + .7*V3 + .7*V4  
           f1 ~~ 1*f1
      
           V1 ~~ .51*V1
	   V2 ~~ .51*V2
           V3 ~~ .51*V3
           V4 ~~ .51*V4
         '



fitted.mod <-
           '
	   f1 =~ NA*V1 + V2 + V3 + V4
           f1 ~~ 1*f1
	   '

reps <- 1000
rho <- double(reps)
alfa <- double(reps)

for (i in 1:reps){

datum <- simulateData(model, 100)
run1 <- cfa(fitted.mod, datum)

Sigma.hat <- fitted.values(run1)$cov
Sigma.error <- inspect(run1,"coef")$theta
rho[i] <- 1-sum(Sigma.error)/sum(Sigma.hat)
alfa[i] <- as.numeric(alpha(datum)$total[1])
}

(sum(alfa>rho)/reps)*100
notice some VERY nice things here. a one-factor, tau-equivalent model HOLDS in the population (that's under the 'model' label). a one-factor model is *fitted* to a sample size of 100 one thousand times and both rho (or omega, or composite reliability or whatever you wanna call it) and cronbach's alpha are calculated, so i end up with 1000 rhos and 1000 alphas.

then notice that i calculate the proportion of times that alpha is greater than rho. my R gave me somewhere around 6.5%. so 6.5% of the time in this simulation alpha is greater than rho. sure, it isn't THAT much.. particularly because it's only 6.5% out of a 1000.... but please keep in mind that we're talking about the most ideal of ideal cases, the case where ALL the assumptions for alpha to either be equal to or less than rho hold! so the point i'm trying to make is that if even for data where all the conditions are met for alpha to be an accurate estimate of rho (or at least a lower bound of rho) you can still find instances where alpha is greater than rho, then it really is expected that, every now and then, alpha would be greater than rho for no other reason aside from sampling variability. now consider the case the OP presents... with TEN factors! and i'm willing to bet my brownies it doesn't even fit the data (by the chi-square test)! tht's why i think CowboyBear hits the nail on the head by saying that, for practical purposes, we don't really know anything about the relationship between alpha and rho when working with real data.
 

CB

Super Moderator
#18
lemme quote from one of the most comprehensive bibles of SEM out there: the EQS Manual (we're going old school, yaaay!). on page 119 where it talks about reliability it says:

Cronbach’s alpha (1951) is well-known. Developed for EQS, rho is based on the latent variable model being run. This could be any type of model with additive error variances. When this is a factor model, this gives Bentler’s (1968, eq.12) or, equivalently, Heise and Bohrnsted’s (1970, eq. 32) omega for an equally-weighted composite. With the /RELIABILITY command, it is based on a one-factor model (Raykov, 1997ab), and then rho is the same as McDonald’s (1999, eq. 6.20b) coefficient omega. Note that McDonald’s omega is a special case of Heise and Bohrnsted’s omega.

this is why i said, for practical purposes, they're all sorta doing the same thing. particularly beacuse reliability is usually thought about within the (hopefully), one-factor, tau-equivalent model.
Ahhh, righto! Part of the reason I got confused is because I'm more familiar with a formula for composite reliability that Raykov developed a bit later, that takes into account correlated errors specified as part of the model:

Reliability = (sum of factor loadings)^2 / ((sum of factor loadings)^2 + (sum of error variances) + (2*sum of error covariances))

(Presuming that the latent factor has been scaled by setting its variance to 1).

I'm pretty sure that the above version of his formula isn't equivalent to coefficient omega, but it's interesting to know that the version without error covariances is equivalent! :)
 
#19
omega... rho.... composite reliability... it goes by many names, with each researcher lending its name (i think it's McDonald's omega, Bentler's rho and Raykov's composite reliability)

Hi spunky,

sorry for joining this old conversation, but this message you wrote let me think very much about the reliability issue.

You said there different names for the same thing like omega, rho and composite reliability. On ething gets me confusing: I am using SMARTpls a sem modeller and this software is telling me both: the rho_A and the composite reliability and they're a bit different in the result. Composite Reliability is higher than the rho_A for every construct. Now my question: are they really the same or does they just measure the same "thing" on a slightly different way?

Your answer will be much appreciated. I hope this topic is not to old and you get an alert that i wrote something here.
Cheers Nylo
 

spunky

Doesn't actually exist
#20
I am using SMARTpls a sem modeller
yeah, i'm gonna stop you right there. this is the problem. the method of Partial Least Squares for purposes of structural equation modeling (SEM) as understood in the social sciences is simply flawed. among its many drawbacks it has the somewhat shameful peculiarity that it models a covariance matrix that does not converge to the population covariance matrix from your sample... so god knows what you're modeling at the end of the day.

the people who advocate this method (mostly form business/information systems) simply took the theoretical developments from covariance-based SEM and plugged them into their software without ever thinking twice whether or not they made sense. the only reason of why i think PLS-based SEM is popular is because Wynne Chin has spent a lot of time time spreading out misinformation about maximum likelihood-based SEM so he can sell his stuff.

so yeah... in conclusion, only the devil knows what SMARTpls is doing here. there exists no mathematical basis behind a theory of reliability coming from the world of Partial Least Squares. i mean, for starters, the estimator doesn't even partition error variance! ughh!

ok, i'm gonna shut up now because i could just go on an on with how many ways PLS-SEM is wrong when used as a substitute for traditional ML-SEM.