Are McNemar's and Binomial the right tests for my data

#1
Hi everyone.

I have performed an experiment, where I have real and synthesised movie data. My participants were shown either a real movie, or a synthesised movie (of talking heads), in a random order, and had to guess whether the movie was real or synthesised.

I have 17 participants, each being shown 120 movies, making 2040 samples.

I got a binomial test result of 0.58 (using matlab binofit). Of 2040 samples, 1189 were guessed correctly which I think shows significant difference.

I also tried a Macnemars test. The confusion matrix is as follows

[506 359]

[449 582]

where top left is 'said synth when synth', top right is 'said synth when real', bottom left is 'said real when synth' and bottom right is 'said real when real'.

Forgive my clumsy notation, statistics isn't my thing.

So anyway, the macnemars test for that gives me 9.91, which again I think shows that there is a significant difference.

Someone has pointed out to me that binomial may not be appropriate, as samples from the same participants cannot be considered independent. Is there any other test I should be using?

Help grateful received!
 

hlsmith

Not a robit
#2
Yes, there is an independence issue. A hierchical model may be appropriate controlling for person level random effects. I will let others chime in with opinions as well, in that there may be a better option.
 
#3
Actually I've just realised that the McNemars test isn't a McNemars just, its just a Chi Squared test. So now that I have the Chi Sq statistic of 9.91, am I correct in obtaining the Phi Coefficient: sqrt(X / N) = 0.06. (where X = Chi Sq)

So 0.06 is a very low correlation right? What test do I use to see if its significant?