A/B testing (experimentation) question

R

raito

New Member
#1
#1
I am new to A/B testing and experimentation. I have 2 questions. Any help is appreciated.

Question 1:
I need to test 3 versions A, B and C. My first option is to run A vs B and the winner vs C. Another option is to run A vs B vs C. What things should I consider when comparing these approaches? Which approach is better and why?

Question 2:
I decide to sequentially launch 10 new changes in a product. I run 10 A/B t-tests in sequence one after other for lift of 5 units each. Assuming, each of the changes gave me a significant lift, I believe in the end I will get a total lift of 50 units. Is this accurate? If not, what things should I be considering here or what more pieces of information do I need?
 
Last edited:
hlsmith

hlsmith

Less is more. Stay pure. Stay poor.
#2
#2
What is the context? How big of samples can you have? What is the turnaround to to getting results? What explicitly is the outcome and how is it measured? Haven't used lift much, but depending on changes having independent or dependent effects you could expect infinite impact variability on outcome (additive, multiplicative, etc.).
 
O

obh

Active Member
#3
#3
raito said:
I am new to A/B testing and experimentation. I have 2 questions. Any help is appreciated.

Question 1:
I need to test 3 versions A, B and C. My first option is to run A vs B and the winner vs C. Another option is to run A vs B vs C. What things should I consider when comparing these approaches? Which approach is better and why?

Question 2:
I decide to sequentially launch 10 new changes in a product. I run 10 A/B t-tests in sequence one after other for lift of 5 units each. Assuming, each of the changes gave me a significant lift, I believe in the end I will get a total lift of 50 units. Is this accurate? If not, what things should I be considering here or what more pieces of information do I need?
Click to expand...
Hi R,

When comparing the two methods, you should consider comparing the following:
1. Time to conclusions
2. Standard deviation
3. Correct significant level (like Bonoferoni)

Also if you compare several measurements, you may want to see the full picture (not only the best version)
Each version may be better in other parameters.
 
You must log in or register to reply here.