+ Reply to Thread
Results 1 to 3 of 3

Thread: How to prove the magnitude of improvement (two data sets)?

  1. #1

    How to prove the magnitude of improvement (two data sets)?




    Hi Guys!

    I am struggling a bit with some statistical validations. Currently using Google Analytics in order to get data from my website. Testing two versions of the website, not at the same time (A/B-test), but I usually test one version for 1 week and measure the conversion rate (defined as number of purchases divided by number of visitors). I then change something on the website and run it for a week and see if this conversion rate goes up or down.

    Made a "two sample t-test" with the null hypothesis that the two sets of data had the same mean and could disregard that hypothesis. So my conclusion is that the change I made on the website has improved it to the better. The measured increased conversion value is 1.3% and the old conversion rate was 0.9%.

    Now to the real question, is there a way to use data to also prove that the improvement is indeed 44% (1.3/0.9) ? The t-test I made only supports the hypothesis that the change I made on the website has made one of the versions better than the other (as I understand it).

    Thanks a lot for any help!

  2. #2
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: How to prove the magnitude of improvement (two data sets)?

    hi,
    your statistical SW should also give you the confidence interval and mean for the difference.Isn't this what you need?

  3. The Following User Says Thank You to rogojel For This Useful Post:

    analytics_stats (10-02-2016)

  4. #3
    Points: 12, Level: 1
    Level completed: 23%, Points required for next Level: 38

    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: How to prove the magnitude of improvement (two data sets)?


    Hi Rogojel,

    I found an online calculator which only provided me with p-value. I will dig in to the two terms you provided me with.

    Thanks!

    //Analytics beginner

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats