I have 10 different groups (of varying sizes). I have recorded 'yes' or 'no' responses from all the members in each group regarding their use of a specific item (these same data were recorded for multiple items in the study). I am trying to determine if any of the groups responded to using the item differently.
I have tried to analyze the groups responses using chi-square, but every chi-square value calculated for the responses to each item is coming back as significant, even when the PROPORTION of each group that has said 'yes' is almost identical for all groups (ex: group 1: 96% responded yes, group 2: 94% responded yes, group 3: 95% responded yes, etc). After some reading, I discovered this is a common problem associated with chi-square for extremely large samples size (that is, everything begins to 'appear' significant due to the increased chi-square value). Here is a document that describes this 'phenomenon' on page 7, if anyone is interested http://courses.ncssm.edu/math/Stat_I...20Analysis.pdf . I have approximately 3000 responses from my groups for each item in question.
Does anyone have any other suggestions on how else I can compare these data to see if some groups responded differently to the use of a certain items? I know there is a "z-test for two proportions", but I'm afraid since I can only compare 2 group proportions at once using this method, and since I will have to repeat the comparison multiple times for all groups, there will be increased error.
I appreciate any suggestions.
Every statistical procedure, and I mean every, has the same problem when sample sizes become large - small differences tend to become significant.....what you'll need to do is determine from a practical standpoint, what size of a difference is "meaningful" and go from there.
I appreciate your help. So, if all groups use an item 90-99% of the time, I may decide to say there is no meaningful difference b/t each groups use of the item.
However, if Groups A thru I use an item 90-99% of the time, but Group J uses the item only 45% of the time, I may decide to say that is a meaninful difference in the usage of the item by Group J. My final question is, besides stating that I decided what constitutes a difference and what does not based on my personal opinion, is there a statistical test that can support my claim.
Not really. It all boils down to how convincing the "evidence" is. A reasonable person would agree with your statements above regarding the absence / presence of a meaningful difference.
This helps. Thanks again for your quick responses!
Posting Permissions
|
|
Reply With Quote