Margins of error - specific case

#1
Say I set a test where all questions that have answers that are right or wrong. There are 10 questions sourced from a limited bank of 50.

I create 3 different tests each with 10 questions and then assign the tests to students at random. Not all students sit the same test but they will have some questions in common.

If I find that 40 students answered a question and scored 70% then I would I work out the margin of error around that 70%. I want to have an idea what the populate proportion is (say 95% CI).

Q.
* Do I treat each individual question as the sample proportion and then do 1.96 * sqrt(0.5*0.5/40) for the margin of error?
* Do I somehow take into account the number of questions in the test of the total number of question
* A combination of both.

I think that the individual questions will effectively be assigned randomly to students, despite the fact it's the tests that are randomly handed out. Another way to think about it is that once the test is assigned there is no choice.

Any advice would be gratefully received!
 
#2
At first sight the number of tests and the size of the test, and the size of the bank don't matter. You are interested in the answers to one particular question for which you have a sample of 40 student answers (out of say 200 students.) The sample proportion is 0.7. The MoE around that is given by 1.96 * sqrt(0.7*0.3/40) = 14.2% approximately for a large population (if your students are a representative sample of the whole country.).
However, if you are interested in just your finite population of 200 students, there is a correction for this. Multiply the MoE given by sqrt((N-n)/(N-1)) or sqrt((200-40)/(200-1)) or about 90% giving a MoE of 14.2%*90% = 12.7% kat
 
#3
Thanks @katxt that is reassuring :)

If I wanted to group 4 of the questions into Test Section A then I'd work out the proportion that were correct. Say I find that on average students get 3 of the 4 questions right. The proportion would be 75%. The MoE is likely to be greater for the questions combined.

Would I take the separate distributions from each question? i.e.
1.96 * SQRT((p1*q1/n1) + (p2*q2/n2) + (p3*q3/n3) + (p4*q4/n4))

q = 1-p

Since there are 3 different tests, in some cases a student would have answered the question and other times not. Does that throw a spanner in the works?
 
Last edited:
#4
I'm sorry, but it isn't clear to me just what you are proposing to do. Does each test have a section A with four questions out of the 50 and the score for section A can be 0 to 4 for each student, and there are 40 students. and say the average is 3/4 or 75%?
 
#5
That’s right, each test of 10 questions has a section A with 4 questions. These come from the bank of 50. The 40 students answer on average 3 of the 4 questions from section A correctly.
 
#6
So, your data looks like this 2 4 3 1 0 3 4 2 3 2 1 ... with a mean of 3.0 and n = 40. This isn't about proportions any more. It's about the mean. The data isn't normal, but you will probably get a reasonable result using the standard xbar +/- t.sd/sqrt(n) formula. t is about 2.02 rather than 1.96 for 40 students.
Alternatively, you can use resampling for another plausible result. kat
 
#7
I see what you mean, I wasn't clear. I'm focusing on the proportion getting all 4 correct, would I apply my formula above using 2.02?

One more question, you've been very helpful :) For section B, I have sometimes 2, sometimes 3 questions, I'd like the proportion of students that answer all questions in section B correctly. Say 1 of the 4 tests has 2 questions in section B and the other 3 tests have 3 questions.

If student's section B has 2 questions then the variation around getting both correct is: (p1*q1/n1) + (p2*q2/n2).............................................(G)
If student's section B has 3 questions then the variation around getting both correct is: (p1'*q1'/n1') + (p2'*q2'/n2') + (p3'*q3'/n3')........(H)

So across all 40 then the overall variation is one component of variance of type G and 3 of type H. Add all together? Doesn’t feel right but maybe it is.

For MoE it would be:
2.02 x SQRT(G1 + H1 + H2 + H3)

Alternatively they’d be pooled into one variance rather than 4 added together.
 
Last edited:
#8
I'm focusing on the proportion getting all 4 correct,
In this case, your data is yes yes no no no yes ... according to whether or not they have all four correct. In this case the answer is a proportion, rather than a mean so you use 1.96sqrt(pq/n) but it is strictly only an approximation. It doesn't work if the proportion gets too close to 0 or 1. (The t = 2.02 stuff is for a mean, rather than a proportion.)

For section B, I have sometimes 2, sometimes 3 questions, I'd like the proportion of students that answer all questions in section B correctly.
Now things are more complicated. The simplest idea is that your data looks like this yes yes no no no yes ... according to whether or not they have all the section B questions right or not. If 80% of the 40 students get all section B right, the moe on the 80% is 1.96sqrt(0.8x0.2/40). This will be a good approximation but it isn't quite right because the probability of a "yes" is not constant.

I don't really get the G, H theory above. Perhaps you could try the GH method and the simpler method just given and see how close they are.
 
#9
* For a single question, we have the MoE as 1.96sqrt(pq/n). Happy with that.
* Section A has 4 questions, measure is % with all 4 correct, this is also 1.96sqrt(pq/n). Think about in yes / no terms really helps and it "chunks" each response. I was trying to incorporate the uncertainty of each individual question but I guess it averages out if we treat each yes / no block as the random sample for each student. i.e. the variation in the proportion handles that.
* Section B has 3 or 4 questions. Using the same logic as above then 1.96sqrt(pq/n) would be applied regardless of the number of questions and the responses would be yes / no. Not sure what you mean about probability of a "yes" not being constant.

Thanks @katxt
 
#10
Not sure what you mean about probability of a "yes" not being constant.
The binomial moe which is the one that you use for a proportion assumes that the probability of a yes/no is the same for each student. I imagined that it would be harder to get 4 out of 4 than to get 3 out of 3. However it is a subtlety that would be unlikely to make a difference in practice, if at all.
 
#12
So, we imagine that all the students sit a supertest of 50 questions. What proportion would get 50 out of 50 right? If we can answer that question, we can put an interval round it as before. However, I can't see a way of answering that question using data from 10 question tests.