+ Reply to Thread
Results 1 to 3 of 3

Thread: Is 3% difference significant in an experiment?

  1. #1
    Points: 4, Level: 1
    Level completed: 7%, Points required for next Level: 46

    Thanked 0 Times in 0 Posts

    Is 3% difference significant in an experiment?

    I am trying to see if certain change in a survey will impact the response rate.

    I sent the survey to 3277 people in the experimental group (with the change implemented) and to 3253 in the control group (without the change implemented).
    In the first group 22% of people filled the survey out, in the second group it was 25%.

    How significant is the difference? What method would you recommend for checking it?

    A little note: I have not much idea about statistics, had classes 15 years ago so if you can explain it in a super simple way, it would be great :-)
    Thank you!

  2. #2
    Points: 3,631, Level: 37
    Level completed: 88%, Points required for next Level: 19
    staassis's Avatar
    New York
    Thanked 41 Times in 39 Posts

    Re: Is 3% difference significant in an experiment?

    In this problem you can use the Z-test for comparing two proportions. The Z-statistic equals

    Z-stat = (p2 - p1) / Standard Error = (p2 - p1) / sqrt(p * (1 - p) * (1 / N1 + 1 / N2)),


    N1 = 3277,
    p1 = 0.22,
    N2 = 3253,
    p2 = 0.25,
    p = (p1 * N1 + p2 * N2) / (N1 + N2) = 0.2349449.

    So we have

    Z-stat = 2.859009,

    which corresponds to the p-value = 0.004249667 << 0.05 = significance level. So the difference is sample response rates is statistically significant and we accept the hypothesis stating that the true response rates are different.

  3. #3
    Points: 12,672, Level: 73
    Level completed: 56%, Points required for next Level: 178
    Master Tagger
    GretaGarbo's Avatar
    Thanked 462 Times in 402 Posts

    Re: Is 3% difference significant in an experiment?

    A somewhat more important problem is that about 75% have not responded. Can you infer anything from such an investigation?

    There was a control and a treatment group. Did you randomize the participants to either treatment or control? If randomization, can you assume that the non-response event is independent of the measured variable?

  4. The Following User Says Thank You to GretaGarbo For This Useful Post:

    rogojel (06-24-2017)

+ Reply to Thread


Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Advertise on Talk Stats