Low Cronbach alpha on recognised instrument

#1
Hi,

I am using a standard survey instrument that during a meta analysis over 15 years has shown a relability of 0.73 for a particular facet. I conducted cronbach alpha for my study and it came back as 0.51 well below an acceptable level. My question is simple, do I disregard the 0.51 as the instrument has been proven already, or do I remove it from the study and explain due to the 0.51 alpha that I have excluded it from any further correlations etc. Or do I continue to use it, but caution any results against it due to the low alpha number.

Thanks for any advice in advance.

Jake
 

hlsmith

Omega Contributor
#2
Reliability typically doe not mean Cronbach alpha, so are you saying 0.73 is the reported Cronbach alpha and the instrument's reliability? Please clarify this before we address whether you are even able to compare your value to 0.73 (might be comparing apples to oranges).
 
#3
The reliability of the OCQ was tested in a meta-analysis by Meyer, Stanley, Herscovitch, & Topolnytsky (2002) with 15 years of OCQ questionnaire results using both the 24 item and 18 item versions of the OCQ. They found the reliability of the OCQ to be .82 for affective commitment, .73 for continuance commitment, and .76 for normative commitment.

here is the narrative for the instrument.

I am just wondering if my cronbach alpha comes in below 0.7 what should I do. Use with caution, remove it from study or go with the facts above. Thanks for replying.

I'm a bit out of my depth here, so any guidance greatly appreciated
 

Karabiner

TS Contributor
#4
I am just wondering if my cronbach alpha comes in below 0.7 what should I do. Use with caution, remove it from study or go with the facts above.
Who can tell without knowing anything about your study
(context, theoretical background, objectives, design,
sampling procedure and sample size, what the scales are
used for...) ?

With kind regards

K.
 

hlsmith

Omega Contributor
#5
Are Affective commitment, continuance commitment, and normative committed groupings of instrument questions? If so, are you grouping the same questions for your 0.73. Also DOES the Meyer et. al paper say reliability, cronbach alpha, or uses them interchangeably?
 
#6
Hi. I'm using SPSS and have grouped the 6 questions of each type (t 18). So cronbach alpha all six AC, then all six NC and then all six CC. Affective came back as .79, Normative as 0.85 but Continuance came back as 0.51. This is an organisational commitment questionaire which I have used in addition to a job satisfaction questionnaire to compare the two and to compare each with demographics if that makes sense. But before I use the information, I elected to run a cronbach alpha as i have seen in other studies/books, but nowhere have I seen what they say you should do if your cronbach comes in below the .7 value. ? So if i am correlating between organisational commitment and job satisfaction should I just use the other two. Is it reasonable/justifiable to remove the continuance aspect based on the low cronbach?
It is used interchangably and to be honest, I'm new to this so some of it going over my head a bit
 
Last edited:
#8
Hi JakeMac,

I apologize if you have done it already, but I have in seen several studies that an unexpectedly low Cronbach's alpha was the result of a data entry/data coding mistake. Have you checked if any specific item of the respective subscale is responsible for dragging down your value?
 

trinker

ggplot2orBust
#9
In the SPSS output there should be a place that says something like Alpha if removed (to go with Regina's comment). If you see this going above the Alpha level this item is suspect. Also be wary of reverse scoring where low is actually high and high is actually low. Many instruments contain reverse scored items that would kill Alpha if you don't enter them in as reverse scored.

If everything is entered correctly then HlSmith's question really is the key to your answer.
 
#10
Hi all, Thank you all for your time.
I have reversed score all the negatively worded items. I have also check the item deleted scale which show no significant change to the figure. There is no mssing data etc. I have checked and rechecked this.

A brief description of what I'm doing
A demographic survey as independant variable
A job satisfaction survey as a dependant variable
A organisational survey as a dependant variable

From this I want to calculate;
the mean job satisfaction of the group under different facets scores and a total job satisfaction mean. All these facets has a cronbach of near or over .9,
the mean organisational commitment of the group which we actuall be a mean for the three different types as opposed to an overal score,
Correlation between the demographics and the facets of job satisfaction,
Corelation between the demographics and the 3 types of organisational commitment,
Correlation between the 3 types of organisational commitment and overall job satisfaction.

The organisational questionnaire is a 18 question instrument. 6 items for each of the 3 types of commitment. Does that help?

Thanks again all

Jake
 

trinker

ggplot2orBust
#11
@Jake I believe that hlsmith's question was more what you plan to do with the info. Is it policy you want to influence or a pilot study. An acceptable alpha is different for these two cases.
 
#12
It is a cross sectional study for a university dissertation and will not be used for policy but will probably be read by management. I suppose I'm asking if the normal accepted value is .7 then what is the procedure for a research paper that the values comes well below this. If an examiner is to look at what I did and say why did you proceed to use the continuance commitment in the research when it only had a reliabilty cronbach alpha of .51 or am I over analysing this. If I was to declare in the research that due to the fact that the cronbach alpha for this aspect is 0.51 that I have decided to disregard it for the rest of the research. I can do this, but is it a correct train of thought if you were to comment on my decision to do so?
 
#14
Very interesting thanks. I have conducted the glb as suggested in your conclusion.
Here is what I got

Reliability Statistics
Lambda
1 .423
2 .544
3 .507
4 .689
5 .534
6 .551
N of Items 6

Not sure if that helps though
I'm far from an expert and may be going further than I need to with this, but learning as I go
 

Lazar

Phineas Packard
#15
I don't think that is the glb. From memory SPSS will not provide the glb. Rather you have to do it in the psych package in R.
 

hlsmith

Omega Contributor
#16
I would still report the alpha in the dissertation, otherwise you are just limiting the readers knowledge. Every instrument performs differently in every sample and everyone implements protocols differently. If your alpha is different that could just be the way it is as well with or without systematic error.
 

CowboyBear

Super Moderator
#17
The reliability of the OCQ was tested in a meta-analysis by Meyer, Stanley, Herscovitch, & Topolnytsky (2002) with 15 years of OCQ questionnaire results using both the 24 item and 18 item versions of the OCQ. They found the reliability of the OCQ to be .82 for affective commitment, .73 for continuance commitment, and .76 for normative commitment.
I've had a very quick look at the paper. It doesn't really matter much, but don't you have the continuance commitment and normative commitment reliabilities around the wrong way? I read the paper as saying .76 for continuance commitment. They don't seem to say what kind of reliability estimates they're talking about, so presumably the meta-analysis is based on a hodgepodge of Cronbach's alpha, test-retest, split half and other reliability estimates.

Most psychometricians view reliability as a property of data, not of an instrument. So the test could well produce "reliable" measurements in some samples, but not in yours, meaning that probably the estimate in your own data is the one that matters most.

But see the excellent article Lazar posted about the limitations of alpha.

My question is simple, do I disregard the 0.51 as the instrument has been proven already, or do I remove it from the study and explain due to the 0.51 alpha that I have excluded it from any further correlations etc.
Avoid the word "proven" when you're talking about empirical research. Proofs are in maths and logic, not empirical science.
 

Lazar

Phineas Packard
#18
I still think SEM is a possible way forward. As measurement error is directly estimated resulting parameter estimates based on the underlying latent are unbiased to said measurement error. Thus a low reliability is less of a problem.
 
#19
Ok gents, thanks a lot for the all your time and assistance. As I have mentioned I'm a bit out of my depth here. O.76 was the CCS, my mistake. The values I have put up are the Guttman Lower Bound values for the six items calculated on SPSS 20 and have confirmed these by viewing the help topics in spss. Does this help? I am really trying to do well on the dissertation, so I want to make sure nothing I do is wrong. Is it fair to say ( unless the glb can be included and helps the situation) that I could eliminate the continuance commitment element from the rest of the research based on the low cronbach alpha that I have attained for my data in that particular aspect. Is this too bold or would other authors have done this? I understand the article about use and misuse of cronbach but for an undergraduate degree dissertation I don't think they can pull me up too much by not going further with it? So if I summarise, and please comment on this, I have used a standard recognised instrument, but for my particualr study one aspect/grouping have returned a lower than acceptable cronbach alpha. Therefore I have decided to remove this aspect/group from the remainder of the research due to this fact. If I go a bit further and include the glb and say that this too has yielded low result confirming that the aspect/group is unreliable for the rest of the study? (of course I'm presuming you need a glb of over .7 too, which could be completly wrong?) HELP. I have to have a draft of this done by saturday night and this has become a sticking point that is getting more time than I hoped.