You should use a chi-square goodness of fit test. The expected frequencies are set by default as equal for all the frequencies. So if some of them are less than or greater than that equal frequency, significantly, it will flag it.
Dear all,
I'm trying to figure out how to best determine if there is a significant differences between how often a particular answer to a question has been chosen.
To be more specific: I have a multiple choice question with 6 possible answers. This question has been asked twice (with different phrasing of the question). I would like to find out if the answers have been chosen with the same frequency, independent from the phrasing). My data consists of the answers selected by each participant.
I googled the web and mostly found references to a chi-square test. But I don't have any expected frequencies which I could use.
Thank you for your help!
You should use a chi-square goodness of fit test. The expected frequencies are set by default as equal for all the frequencies. So if some of them are less than or greater than that equal frequency, significantly, it will flag it.
Thanks, for your quick reply.
Ok, so I don't have to know the expected frequency, that's good. The examples I found on the web usually only have the frequency for one variable but I have multiple frequencies.
For each subject I have something like:
where A1, A2, .... are the answers which have been selected.Code:Phrasing 1 Phrasing 2 Subject 1 A1, A3, A6 A1, A2, A3
I can calculate the frequency for each answer and compile a table like:
How can I use chi-square in such a setting?Code:Phrasing 1 Phrasing 2 A1 10 20 A2 15 27 A3 11 11 A4 20 10
Thanks for your help, I really appreciate it!!
Actually I coulnd't understand your setup. But you need to use a chi-square goodness of fit test, not a usual chi-square, if you don't have a setup in which the expected variables are to be observed.
Chi-square goodness of fit works like this: you give it some frequencies, for example you have one question with 5-choice answers and 1000 subjects had answered to it. So the frequency of each of those choices is for example 5%, 20%, 35%, 18%, and 22% (answered by the respondents, for example 50, 200, 350, 180, nd 220 participants have chosen the answers 1, 2, 3, 4 and 5, respectively). Now you run a chi2 goodness of fit test. It counts the variables (=5) and calculates the expected frequencies (= 100/5 = 20%). Now compares the observed percentages with these percentages: 20%, 20%, 20%, 20%, and 20%. It then decides which observed frequency is significantly greater or lower than a 20% frequency. I think now you can find a way to fit your data with this method.
In my opinion, you can either compile the questions, or even assess the answers to each question. The latter needs several chi2 GoF tests (each for one question).
Thanks for your explanation, things are getting clearer
But I'm not sure if we are talking about the same thing. What I would like to find out if there is a significant difference between the frequencies of the answers in regard to the phrasing (one question is asked twice with different wording), in other words, have they selected the same answers with the same frequency regardless of the wording.
Thank you very much!
You mean you need to know if by changing the phrase, the people still answer the same choices? Please explain more in a more details and easier way to understand. I am not sure I know what is your exact problem, what is your exact question, and what is your exact design. But would be glad to help if I could.
Yes, thats exactly my question. I'm sorry for my confusing explanation. The questionnaire contained 5 multiple choice question each asked twice. Answers were the same for each phrasing. I would like to know if the people selected the same choices. I could perform the test for each of the 5 questions separately, that would not be a problem.
Data for Question 1 looks for example like that:
where 1 means the answer has been selected and 0 means not selected.Code:Phrasing A Phrasing B A1 A2 A3 A4 A5 A6 A1 A2 A3 A4 A5 A6 Subject 1: 0 1 0 1 0 0 0 0 1 1 0 0 Subject 2: 0 1 1 1 0 0 0 0 1 1 0 1 ..... Subject 59: 0 0 1 1 0 0 0 0 1 0 0 1
I then calculated how many times each answer has been selected, which gave me something like
What I basically would like to know is, if Phrasing A and Phrasing B can be considered to be equal. My idea is, if the same answers have been chosen with the same frequency for Phrasing A and Phrasing B I could say that the two phrasings are equal.Code:Phrasing 1 Phrasing 2 A1 10 20 A2 15 27 A3 11 11 A4 20 10
Last edited by poons; 08-30-2012 at 07:02 AM.
Well that is much clearer now
I see in each question there is A1 to A6 (meaning that there was 6 choices not 5), besides I see each participant has picked up two answers in each question instead of one (because there is two 1s in each question in each phrasing column [A/B]). So I can't say I have understood it perfctly but might answer to some extent. And I don't know why A1, A2, A3 in the column B (but this is not important IMHO).
If you want to compare these frequencies you can run a 2 x 6 chi-square test (usual chi-square), The expected values are provided by column A and the observed values are provided by column B (or vice versa, the calculations is the same). Of course we are assuming that A1 corresponds B1, and A2 corresponds to B2, and A6 corresponds to B6. But if you have also shuffled the choices in the second column (column B) you should first find which choice in A is a counterpart to a choice in B.
You might also compare the distributions of the answers in column A and B. A Kolmogorov-Smirnov test might be used for this purpose.
Ok, that compiling thing is correct. Then compare those compiled lists of numbers in a 2 x 6 table, using a usual chi-square (not the goodness of fit test). But once you said 5 multiple choices, then you said 6 choices (A1 to A6), then you said 4 choices (A1 to A4)? Please be precise when describing your design to make sure nothing is missing. I think that is a typo. If so, you could compare the compiled tables as you had guessed.What I basically would like to know is, if Phrasing A and Phrasing B can be considered to be equal. My idea is, if the same answers have been chosen with the same frequency for Phrasing A and Phrasing B I could say that the two phrasings are equal.
Note that you have to put real numbers in the table, not theor percentages, in order to get a correct answer.
poons (08-30-2012)
Thank you, that makes it clear!!
What I meant was I have 5 multiple choice questions, each having 6 possible answers.
You really helped me a lot. I appreciate it.
After re-reading your post again, I still think we have some communication difficulties.
What I have posted above is the data for just 1 multiple choice question. A1 ... A6 are the possible answers to this question. Once for phrasing A and once for phrasing B. Each subject can have selected up to 3 answers. For example, subject 2 selected A2, A3 and A4 for phrasing A but only two answers (A3,A4) for phrasing B.
But I guess the 2x6 chi-square would be valid. However, as far as I remember the sum of observed values has to be the same as the sum of expected values. Which is not the case in my scenario, because one and the same person could select only one answer in phrasing A but three answers for phrasing B.
I really appreciate you time and your help.
But this is exactly what I understood
This is new data again! I hadn't noticed it if you had told it before.But I guess the 2x6 chi-square would be valid. However, as far as I remember the sum of observed values has to be the same as the sum of expected values. Which is not the case in my scenario, because one and the same person could select only one answer in phrasing A but three answers for phrasing B.
I think the chi-square still works OK. Because I think it is not necessary for the rows of the contingency table to be of the same total size. Besides, you can run a KS.
"victor is the reviewer from hell" -Jake
"victor is a machine! a publication machine!" -Vinux
Hi,
may be that there is a problem with the use of chi-square test, in that we are dealing here with non-independent proportions (ie, proportions are based on the same sample of subject).
May be I am wrong, but perhaps something like McNemar test should be used. The only problem is that it works for 2x2 tables. So, you should reshape your data in order to perform the test. May be this page could give you some ideas on how do it.
Hope this help
Gm
http://cainarchaeology.weebly.com/
Please note that by "reshaping" I mean to build a 2x2 cross-tab where phrasing choices are tabulated against each other: phr. A (yes/no) vs phr.B (yes/no).
Gm
http://cainarchaeology.weebly.com/
Thanks GM That would be great to exploit the paired nature of data However, it might compromise the main question, which is "are these questions the same in essence?". I think if we dichotomize 6 choices into 2 choices, we will lose a great deal of data. For example all scores A1, A2, or A3 will be considered the same, and A4, 5, and 6 will considered the same too. So we can't confidently talk about the result of our comparison, if it becomes nonsignificant (which is the favorable results). For example, if we compare the phrasing A with B (in a dichotomized setup), and conclude that there is no difference between the two groups, we don't know whether if we had not aggregate the scores, we would again see a similarity between the two groups each with 6 choices? Nut am not sure, maybe there is a way to avoid this problem
"victor is the reviewer from hell" -Jake
"victor is a machine! a publication machine!" -Vinux
Hi,
yes you are right. But, in my opinion, the main issue is the non independent nature of the data.
Besides, it seemed to me that the main focus was on the phrasing.....
Anyway, this is an interesting thread and I look forward to read more replies or alternative suggestions.
Thanks
Regards
Gm
http://cainarchaeology.weebly.com/
victorxstc (08-30-2012)
Tweet |