# Thread: Is 3% difference significant in an experiment?

1. ## Is 3% difference significant in an experiment?

I am trying to see if certain change in a survey will impact the response rate.

I sent the survey to 3277 people in the experimental group (with the change implemented) and to 3253 in the control group (without the change implemented).
In the first group 22% of people filled the survey out, in the second group it was 25%.

How significant is the difference? What method would you recommend for checking it?

A little note: I have not much idea about statistics, had classes 15 years ago so if you can explain it in a super simple way, it would be great :-)
Thank you!

2. ## Re: Is 3% difference significant in an experiment?

In this problem you can use the Z-test for comparing two proportions. The Z-statistic equals

Z-stat = (p2 - p1) / Standard Error = (p2 - p1) / sqrt(p * (1 - p) * (1 / N1 + 1 / N2)),

where

N1 = 3277,
p1 = 0.22,
N2 = 3253,
p2 = 0.25,
p = (p1 * N1 + p2 * N2) / (N1 + N2) = 0.2349449.

So we have

Z-stat = 2.859009,

which corresponds to the p-value = 0.004249667 << 0.05 = significance level. So the difference is sample response rates is statistically significant and we accept the hypothesis stating that the true response rates are different.

3. ## Re: Is 3% difference significant in an experiment?

A somewhat more important problem is that about 75% have not responded. Can you infer anything from such an investigation?

There was a control and a treatment group. Did you randomize the participants to either treatment or control? If randomization, can you assume that the non-response event is independent of the measured variable?

4. ## The Following User Says Thank You to GretaGarbo For This Useful Post:

rogojel (06-24-2017)

 Tweet

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts