Yes/No Questionnaire Analysis

[Akelly1]

Hey all. I'm having issues trying to figure out how to record and analyze some data I've collected. Here's the short story:

8 kids took before and after yes/no questionnaires and answers are scored with 1 or 2 points. Questions are phrased so that sometimes Yes is worth 2 points and sometimes No is worth 2 points (to control for response set). The questionnaires do not have the same questions and do not have the same NUMBER of questions.

I've considered a simple t-test for two correlated samples. This would certainly be the easiest to do by transposing the qualitative data to quantitative by "grading" as points earned out of total possible points. Does the fact that participants were children brought by their parents to a presentation at a zoo change that? Is this considered true random sampling? Is the "grading" method considered an equal-interval scale? If so...

I've considered the Wilcoxon signed rank test, but how would I enter this? Could I "grade" the questionnaires and enter them as before/after values? Does the nature of the questionnaire have any bearing on this? If so...

I've considered the McNamar test, but how would I score this? Would I have to do it for each individual participant with 0 for responses earning point and 1 for responses earning 2 points? How would I combine the data to look at the whole group?

And, on a more general note, I'm not sure I totally understand the difference between what the Wilcoxon signed rank test and the McNamar test tells us. Would someone be willing to explain it in a "Stats for Dummies" way? It's not completely foreign, but stats are not my forte.

Thanks for any and all help.

[CowboyBear]

Hi there.

8 kids took before and after yes/no questionnaires and answers are scored with 1 or 2 points. Questions are phrased so that sometimes Yes is worth 2 points and sometimes No is worth 2 points (to control for response set). The questionnaires do not have the same questions and do not have the same NUMBER of questions.

Can you expand on what you mean about the questionnaires not having the same questions, or number of questions? This seems pretty crucial. If the two questionnaires are asking about different things, I don't know how meaningful it would be to look for a change in mean or medians from time 1 to time 2. It might be helpful to know a little about what the actual aims of your analysis are.

Is this considered true random sampling?

Nope, it's probably better described as a convenience sample.

[Akelly1]

The before and after questionnaires were created by dividing another questionnaire, the Primary Attitude Scale (PAS), in half. Therefore the questions on the before questionnaire are not the same ones on the after questionnaire. The questionnaire was created by the Humane Society to score kids' humane attitudes toward animals. There are an odd number of questions on the PAS, and therefore the before and after questionnaires have a different number of questions.

Annnd...since it's not random sampling, t test is out?

I realize it's a strange way to go about creating the questionnaires, but that part was out of my hands. I don't think my supervisor realized what she put me up to...and neither did I.

[CowboyBear]

But what's the aim of the research? What are you actually trying to find out here?

[Jake]

Like CowboyBear hinted at, the fact that the time1 and time2 questionnaires contained zero overlapping items is a fundamental problem. Even when/if you settle on some statistical procedure and then run the analysis, there is simply no way to tell whether any differences that you may see would be due to something of interest that happened between time1 and time2, or instead just to the items on one of the questionnaires being easier to endorse overall compared to the items on the other questionnaire.

If the questionnaires contained some overlapping items... even just a few... there are techniques for using this overlapping information to make comparisons between the two sets of scores meaningful. But with zero overlapping items, I hate to say it, but I think you are S.O.L.

[Akelly1]

I figured there'd be lots of issues, but this is what I have to work with. I'll have to mention all of that in the write up. Until then, what are your suggestions for tests? I need to start somewhere.

I'm trying to determine whether a zoo presentation had an effect on humane attitudes.

[Karabiner]

You could calculate the proportion of "humane" answers before
and after and use the Wilcoxon signed rank test.

Kind regards

K.

[CowboyBear]

I'm trying to determine whether a zoo presentation had an effect on humane attitudes.

Every research design has strengths and weaknesses, but at a certain point I think it's reasonable to say that you just do not have sufficient or appropriate data to answer this question. As Jake has said, the fact that different items were used at times 1 and 2 mean that you simply cannot determine whether a difference in scores between the two time points is because of a change in attitudes, or simply because the items are different. On top of that your sample size is extremely small, and the lack of a control group means that you can't rule out any of the major threats to internal validity.

My best advice is honestly to go back to your boss and say that s/he needs to provide more appropriate data for you to work with: At the very least, you need a larger sample of kids who have filled out the whole humane attitudes survey before and after the zoo visit. On top of that, figuring out a control group of some kind would be pretty important. I know that isn't the advice you're looking for, but hopefully it will be helpful in the long run.

[Karabiner]

We sometimes have to use parallel versions of the same test
if we want to determine whether something has changed between
two measurements. Maybe splitting the attitude scale in halves
is roughly the same.

Kind regards

K.

[CowboyBear]

Hi Karabiner, that's a good point. The thing is that there's usually quite a process to go through before a test developer can call two versions of the same test parallel. I.e. we might want the true score variance and error score variance to be the same. You'd also want the mean score in any subpopulation to be the same across the two tests, unless you have separate norms, I'd guess. In this case we don't really know that these conditions hold. Perhaps it would help if at a minimum the scores on the two halves of the test were adjusted to take into account the differing number of items in each half (OP seemed to suggest this was the case).