Item analysis - multiple tests ... or not?

[hlm]

Hi all

I have a number of about 200 items that were responded to by 3 different groups and I want to find differences between the subject groups on these items (or combinations of the items).

I have tried different ways of combinations and explored discriminant analysis, MANOVA, and PCA, as well as graphical solutions.

Now someone suggested I should do comparative ANOVAs on each item first as a way of exploring group differences. I was always told that multiple variables need to be tested together, hence am very hesitant, especially as I expect intercorrelations between variables. On the other hand, my MANOVA is powerless with this large number of variables. I know I could do Bonferroni corrections, but again given the number of my variables, my alpha would be extremely small for each ANOVA.
However, my advisor said that for exploratory analysis (as it is the case here) one is justified to run uncorrected multiple tests. I have seen it in some publications but always considered it bad practice.

I like to have your opinion on that matter and ideally some back-up from the literature? Happy about every comment!

Thanks a lot!
hlm

[CowboyBear]

However, my advisor said that for exploratory analysis (as it is the case here) one is justified to run uncorrected multiple tests. I have seen it in some publications but always considered it bad practice.

I don't really agree with this. In exploratory research, the prior probability of some given relationship being "true" is if anything lower than in confirmatory research (where there is a specific reason for expecting a relationship). See the article Why Most Published Research Findings Are False (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/) for a demonstration of how the probability of a significant result reflecting a true relationship can in fact be very low when the prior probability of the relationship being true is small. I.e. if the number of true relationships vs non-relationships in some study area is very low, a significant finding is most likely false. Using a more permissive approach to p value interpretation with exploratory research compounds this problem; if anything I think one should do the opposite (lower alpha values for exploratory research).

I don't really know what the best analysis method would be for your study. What is the actual purpose of the study? What are you trying to achieve by looking at these group differences?

[hlm]

Hi Cowboy bear

thanks for your response and confirmation of my 'bad practice' assumption :-).

It is an offender study, so I am looking at the difference between three offender groups to identify on which variables the offenders differ the most (which would guide their treatment goals). These variables can be sorted into subgroups, such as lifestyle, criminal history etc.

Ideas?

[CowboyBear]

Hi Cowboy bear

thanks for your response and confirmation of my 'bad practice' assumption :-).

All good!


It is an offender study, so I am looking at the difference between three offender groups to identify on which variables the offenders differ the most (which would guide their treatment goals). These variables can be sorted into subgroups, such as lifestyle, criminal history etc.

Ah, interesting. Well, as I guess you know, there is a lot of existing research into risk factors for various types of crime. One option could be to go with a more confirmatory approach where you specify predictors based on prior research/theory, and test their ability to discriminate between offender groups (e.g. via multinomial logistic regression). You might want to restrict your focus to "dynamic" (changeable) risk factors as opposed to "static" (unchangeable, historic) risk factors, since treatment can't change static risk factors like the nature of the index offence, prior number of convictions, etc.

Another idea - you have 200 items, but presumably to some extent these are divided into certain scales and subscales measuring particular variables? Summating items into particular scales/subscales would cut down the number of variables, making your analysis more manageable. Again, you could use these scale/subscale variables in a multinomial logistic regression to see whether they differentiate between the offender groups.