Cronbach's alpha: Same scale, multiple scenarios... special calculation needed?

[Eaners]

Hi everyone, I'm working on data for my dissertation and had a question regarding Cronbach's alpha.

In a pretest, I asked just over 100 subjects to rate their perceptions of privacy of nine different environments. The scale used for each of the nine environments was the same six-item, five-point Likert-type scale.

I have calculated Cronbach's alpha for the scale in each of the nine environments (range is between 0.51-0.68), and my outside committee member (a psych guy) says that I should just simply report that range.

However, something about that answer bothers me a bit. I would assume that because the same scale was used for all nine environments, there should/would be some way to collect these together in some way. To use a metaphor, you could use the same ruler to measure the height of plants without having to wonder if the ruler was properly calibrated.

... or am I barking up the wrong tree? Would averaging these numbers produce anything meaningful? Or is there some other procedure that I should perform altogether that would make more sense in this situation?

 

[spunky]

would test-retest reliability coefficients serve your purposes?

if not, an average of alphas may or may not sound too crazy depending on the assumptions that you're willing to make for your test. alpha is itself an average (of all possible split-half reliability coefficients for a given test). averaging across alphas would assume that the (hypothetical) sampling universe from which the items were taken are invariant under testing conditions (i.e. each one of the 9 scenarios bears no influence in the reliability of the scales. the differences in alpha you observe arise because of sampling variation and if you could hypothetically calculate alpha in the population of items, each alpha on each of the 9 scenarios would be the same). HOWEVER, if you assume that the items should operate differently as a function of the 9 scenarios then you would indeed be averaging across apples and oranges because there is a scenario X item interaction (in an ANOVA-like context)

 

[Eaners]

I don't think that test/retest reliability makes sense in this case because I only did one trial with the hundred-some subjects.

However, I think I get what you're saying about the alpha average, and I think you're right on. I never assumed, nor did I want to assume, that the environments/scenarios had any bearing on the scales; in fact, I was (all along, and perhaps wrongly) thinking of this six-item measure as a sort of "ruler" that would sort of "poop out" a nice-and-neat 1-to-6 score that corresponds to perceptions of privacy. In other words, my thinking was that I could/should be able to insert essentially any environment, ask the same items, and (if alpha is satisfactory) be able to say "this is X environment's score."

I suppose I was mainly curious if averaging Cronbach's alpha was something that "is done," mainly since this is for dissertation purposes and I didn't want to have to argue a relatively picayune point, particularly since it's part of a pretest.

 

[spunky]

I don't think that test/retest reliability makes sense in this case because I only did one trial with the hundred-some subjects.

well, what i understood of your design (correct me if i'm wrong) is that the scale was administered repeatedly in various scenarios, right? so do you have the same people answering the same scale under 9 different conditions? like:

Person 1 answered Items 1-6 in Scenario 1, Person 1 answered items 1-6 in Scenario 2, Person 1 answered items 1-6 in Scenario 3...

OR does it look more like

Person 1 answered Items 1-6 in Scenario 1, Person 2 answered items 1-6 in Scenario 2, Person 3 answered items 1-6 in Scenario 3... and so on.

i just want to see whether you have people crossed with items and scenarios (so every person responds every item in every scenario) OR if you have people nested within Scenarios (so only certain people respond to all items within each scenario). either or, i think you can see that even though you don't have repeated administrations of the test over time, you have repeated administrations across Scenarios so test-retest reliability is not completely out of the question.

However, I think I get what you're saying about the alpha average, and I think you're right on. I never assumed, nor did I want to assume, that the environments/scenarios had any bearing on the scales; in fact, I was (all along, and perhaps wrongly) thinking of this six-item measure as a sort of "ruler" that would sort of "poop out" a nice-and-neat 1-to-6 score that corresponds to perceptions of privacy. In other words, my thinking was that I could/should be able to insert essentially any environment, ask the same items, and (if alpha is satisfactory) be able to say "this is X environment's score."

I suppose I was mainly curious if averaging Cronbach's alpha was something that "is done," mainly since this is for dissertation purposes and I didn't want to have to argue a relatively picayune point, particularly since it's part of a pretest.

well, i've seen alphas been averaged but it's usually across more homogenous situations... i feel like i would need to know more about your design (like the items, what they're attempting to test, the Scenarios you're describing, etc.) to make a better assessment. the fact that you mention "this is X environment's score." makes me think you may implicitly want your items to behave differently accross scenarios... or maybe not. not sure how much you want to dwell into this, though. issues of reliability are usually a Pandora's Box within Psychometrics and the more you dig the more convluted things become. i honestly think your test design is not as straight-forward as you may want to think but then again i do this for a living so finding minutiae and making a big deal about them is sort of my thing XD

 

[Eaners]

well, what i understood of your design (correct me if i'm wrong) is that the scale was administered repeatedly in various scenarios, right? so do you have the same people answering the same scale under 9 different conditions? like:

Person 1 answered Items 1-6 in Scenario 1, Person 1 answered items 1-6 in Scenario 2, Person 1 answered items 1-6 in Scenario 3...

Yes, this is exactly what it looked like. Same ~111 subjects answered all six items for each of nine scenarios.

the fact that you mention "this is X environment's score." makes me think you may implicitly want your items to behave differently accross scenarios... or maybe not.

In so many words, I needed to first establish that three environments are viewed differently in terms of privacy (something along the lines of "very private," "very public," and "somewhere in between")--thus the pretest. The six-item scale was derived from a couple of definitions of environmental privacy, so ideally a high score on the scale would indicate something akin to "perceived as very private" whereas a low score would indicate "perceived as very public."

Does this make sense?

 

[spunky]

In so many words, I needed to first establish that three environments are viewed differently in terms of privacy

well, it seems like the crux of your issue is here... if the 3 environments are supposed to be perceived differently right from the start it would imply to me then that you're forcing the construct you're measuring (privacy perception) to be multifaceted differently across scenarios... which means you have methods effects per scenario...uhmmmm...

ok, no. stick to the range of alphas then as the "psych guy" proposed. you have too many dependencies to account for here and unless you can estimate a reliability coefficient from a Structural Equation Modelling (SEM) perspective, you are averaging across apples and oranges. if you're familiar with SEM you can fit a multi-factor model with a latent variable for privacy perception and various methods effects per scenarios. if everything i just said sounds alien to you, then just report each alpha individually.

this is, of course, contingent once again on how statistical savvy your committee is. if they aren't, i guess you can get away with it (although it is technically wrong).

 

[Eaners]

Got it. The member of my committee from psychology DID write a SPSS textbook, so perhaps I'll just go with his (and your) recommendation to stick with the ranges. Thanks!