# Thread: When does a sample cease to be random

1. ## When does a sample cease to be random

I never thought about this before I got it today. We have low response rates, perhaps 20 percent. Is there a point at which very low response rates make a sample that was random initially (when you picked people to sample) not random? I have not seen that addressed.

Another question is what is the best definition of response rate. Many of the people who are to be sampled are not reached because the contact information is wrong. The company we work with, citing what they say is the public relations norm, argue that the response rate should be number returned/percent called (even if they do not reach the customer in question). I do not know what the old standard was, but my second question is, is that a good way to calculate response rate.

2. ## Re: When does a sample cease to be random

what the F*CK, after years and years you are gonna come at us saying you have a sample not the population?

3. ## The Following User Says Thank You to hlsmith For This Useful Post:

ondansetron (08-02-2017)

4. ## Re: When does a sample cease to be random

You can calculate the response rate anyway you want. You just have to report the method used so that others can interpret it's meaning and what effect it will have on interpretations/generalizations.

Example:
population = all men in a town;
Survey method: mailing;
Randomly sample 20% of males in town (population) to send a survey.

Well you can exclude those people (invalid addresses) and your sample is no longer a random sample of the population, it is a random sample of people with a valid address. This means you can't generalize results to the overall population members unless you know their address status (and it is valid).

Response rate: # of surveys completed and returned / # of surveys sent out # minus those returned to sender because of invalid address.

Now the eligible sample is random for those with a valid address, but based on the characteristic of respondents, it may no longer be a random sample of those with a valid address if the mechanism for who replies is not random. If there is an underlying pattern in response characteristics, some folks have a higher propensity for replying and that mechanism then you can only generalize to those types of people.

It is all pretty intuitive, if you also think about the mechanism for who responds like the missing data problem, it can be reason for replying can be MCAR, MAR, or NMAR. This last line should help you out. So if MCAR you are fine generalizing to people with a valid address, if MAR - you may be able to weight responses to make generalizations to people with valid addresses, if NMAR you are hosed.

5. ## The Following User Says Thank You to hlsmith For This Useful Post:

noetsi (08-08-2017)

6. ## Re: When does a sample cease to be random

Originally Posted by hlsmith
what the F*CK, after years and years you are gonna come at us saying you have a sample not the population?
Different data sets. I run whole populations. This is different data that we do not collect which is being sampled. I was asked for my "expert" opinion on this although obviously I am not an expert.

7. ## Re: When does a sample cease to be random

Well you can exclude those people (invalid addresses) and your sample is no longer a random sample of the population, it is a random sample of people with a valid address. This means you can't generalize results to the overall population members unless you know their address status (and it is valid).
This goes to the heart of my question. If randomness, as compared to bias, is tied to how you initially sample or how customers respond. I understand that responses at different rates will bias a sample and make it of questionable value. The question I was asked was if that would mean the sample itself was not valid, a different question I have never seen addressed.

Personally I think near all non-response rate is MNAR and this is ignored because if you don't you can not complete analysis.

8. ## Re: When does a sample cease to be random

I don't get your question then.

Say you have a population of interest, then you take a random subset to constitute the eligible sample you are going to survey. Next people respond. If the respondents don't represent the eligible random sample then you don't have a random sample! And that effect isnt eneralizable to the random sample without say weighting.

If you are trying to confuse this with a random variable, well reported data from respondents can still constitute a random variable since the sample still has a data generating function for a phenomenon of the represented respondents,which wouldn't be generalizable to the random sample but to the sample represented by the actual respondent sample.

Unless you phrase you question differently I can't help beyond the above posts!

9. ## Re: When does a sample cease to be random

The question really is, what is randomness and when do you not have it or what can eliminate randomness(as compared to having bias which is not what the person asking meant, they meant is the sample no longer random period). This may be too basic an issue to answer and I have never seen it addressed. I think Greta Garbo argued that a sample that is done randomly initially meant that the sample was always random, although it could be biased.

The question asked was simply "We did a random sample and we got very low response rates 9and used a specific definition of response rate, the one in my first post). Does that make a sample non-random (as compared to making it biased). I think it depends on what it means to be random.

10. ## Re: When does a sample cease to be random

You have a population and you take a list of random people you want to contact from it. Now not everyone is going to respond that is in that list, you can get a random response from the random response list. There are near infinite random realizations that you could get of combinations of respondents from that list. Now, you can't do the direct enumeration since there are so many possible groups of realization of respondent from the list, so you can do an approximation. So take your random list and see how many times its characteristics are comparable to your actual respondents, when taking many many random samples from it, say 1,000,000.

PS, if it was a "random" sample, but the response rate was low it isn't biased because it is small, but it may not be representativeˇˇ!!!!!!!!!!!!!!!!!!!!!!!!!!!!ˇ!!!!!!!!!!!!!!

I can flip a coin five times and get all heads, my sample is too small to make generalizations if I have no more info. No biased just too small to provide info on the data generating function. When is it too small, but not biased, well when it doesn't represent the true population, but by chance that is a risk. Simulate to see how often that is a risk. Much like a sample size calculation, state what you what to compare and calculate how many people are needed to prove it if the hypothesis is true. Your hypothesis is that the sample has the sample average characteristics as the population, heck you could use a one-sample tests against constants to answer this question as well.

11. ## Re: When does a sample cease to be random

This paper should be of interest:

http://journals.lww.com/epidem/Abstr...tcomes.13.aspx

12. ## The Following User Says Thank You to hlsmith For This Useful Post:

noetsi (08-11-2017)

 Tweet

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts