Sampling Q - why can't we replace non-responders by randomly selecting new ones?

Modamichelle

New Member
We are planning a survey of schools to get state estimates of substance use among youth. We are doing a two-stage survey, where the first stage is a PPS random selection of schools, and the second stage is random selection of classrooms (then the students in the classroom are surveyed).

We are concerned that a lot of the schools won't respond to the survey. My colleague thinks it would be a good idea to replace those who don't schools who don't respond by randomly selecting new ones.

This is a bad idea, will you help me explain why in a convincing somewhat technical fashion? I tell her it would lead to biased results because those who respond will have a higher probability of being in the sample than those who don't, but she argues it's already biased because we are missing some who do not respond. I think we would just be adding sampling bias on top of non-response bias. Please help!

hlsmith

Less is more. Stay pure. Stay poor.
Please try to reword this sentence:

"My colleague thinks it would be a good idea to replace those who don't schools who don't respond by randomly selecting new ones."

Modamichelle

New Member
"My colleague thinks it would be a good idea to replace those schools who don't respond by randomly selecting new ones."

For example, say we randomly sample 100 schools from a population of 1,000 schools. Say only 60 of the 100 respond to the survey. My colleague wants to randomly sample another 40 schools from the population to try to make up for those who don't respond.

Modamichelle

New Member
I should add that we needed a certain response rate to calculate weighted state estimates. So, assuming increasing the sample size to 140 won't increase the response rate, then I don't see the point. I think she's suggesting that we treat the 2nd wave of random sample as part of the first. So, using my example above, if 20 out of 40 respond in the second stage, then the sample size is 80 out of 100 instead of 60 out of 100.

hlsmith

Less is more. Stay pure. Stay poor.
There are many approaches, but the big thing is understanding who is opting not to participate and whether they differ from those who opt to participate. If non-participation is not a random thing and is systematic, then biases can occur.