Chi square ok for questionnaire responses?

[BJ82]

Hi,

A friend has asked me for some help with statistical analysis of responses to a questionnaire that she conducted as part of her dissertation. I don't want to give her bad advice, so I'd be really grateful if someone could reassure me that my suggestion isn't a statistical abomination!

It's a fashion questionnaire, and there are 10 questions are of the form "How satisfied are you with criteria X?", with a choice of responses between 1 and 7, where 1 is 'not at all satisfied' and 7 is 'very satisfied'. She asks each person the same set of questions twice over - once with regards to specialist online retailers, and once with regards to high street/chain store retailers.

To check if there is a significant difference between the scores for online retailers vs high street retailers, I suggested she do a chi square test - count the number of 1s, 2s etc that the online retailers received from all the questions and compare this to the scores for the high street, and see if the frequencies depart from what you'd expect if there was no difference between the two. I've attached a jpeg showing what I've done to calculate this.

It seems ok to me, but I'm concerned that some aspect of the data might violate the requirements of the chi square test - independence, for instance, as many people will have offered opinions on both online and high street?

Does the chi square seem reasonable for this data? Any assistance would be hugely appreciated :)

[TheEcologist]

I don't think a ch-squared is appropriate:

1) The samples are not strictly independent (but paired).
2) She has ordinal variables (chi-squared is designed for nominal (or 'categorical counts')). Although you could treat it like you do, the questionnaires we're not really designed like that (so its not a robust scientific approach to do this).
3) you want to find out if: 'if there is a significant difference between the scores for on line retailers vs high street retailers'. I would interpret this as: does the mean/median score of A differ from B. Chi-squared goodness of fit is not really the test that answers this question.

Therefore I feel that a non-parametric paired test like a paired Wilcoxon test is more appropriate, it comes much closer to the essence of what your friend wants to know.

hope this helps,

Refs:
http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test

[BJ82]

Thanks for your response, TheEcologist, really appreciate it :)

I did originally try a Mann Whitney test, ranking the mean scores for each question, classified as either online or retail, but it seemed a bit unfair to do so because the decimals of the means made the ranking too easy (if that's clear...). The problem with doing a Wilcoxon, if I'm understanding it right, is that a fair few respondents had never shopped online and so she has a lot of unpaired scores for the retail questions. I got a message today that she's got a book out dedicated to questionnaire analysis, so I'll have a gander at that. Thanks again for your help!