I have performed an experiment, where I have real and synthesised movie data. My participants were shown either a real movie, or a synthesised movie (of talking heads), in a random order, and had to guess whether the movie was real or synthesised.

I have 17 participants, each being shown 120 movies, making 2040 samples.

I got a binomial test result of 0.58 (using matlab binofit). Of 2040 samples, 1189 were guessed correctly which I think shows significant difference.

I also tried a Macnemars test. The confusion matrix is as follows

[506 359]

[449 582]

where top left is 'said synth when synth', top right is 'said synth when real', bottom left is 'said real when synth' and bottom right is 'said real when real'.

Forgive my clumsy notation, statistics isn't my thing.

So anyway, the macnemars test for that gives me 9.91, which again I think shows that there is a significant difference.

Someone has pointed out to me that binomial may not be appropriate, as samples from the same participants cannot be considered independent. Is there any other test I should be using?

Help grateful received!