I am new to A/B testing and experimentation. I have 2 questions. Any help is appreciated.
Question 1:
I need to test 3 versions A, B and C. My first option is to run A vs B and the winner vs C. Another option is to run A vs B vs C. What things should I consider when comparing these approaches? Which approach is better and why?
Question 2:
I decide to sequentially launch 10 new changes in a product. I run 10 A/B t-tests in sequence one after other for lift of 5 units each. Assuming, each of the changes gave me a significant lift, I believe in the end I will get a total lift of 50 units. Is this accurate? If not, what things should I be considering here or what more pieces of information do I need?
Question 1:
I need to test 3 versions A, B and C. My first option is to run A vs B and the winner vs C. Another option is to run A vs B vs C. What things should I consider when comparing these approaches? Which approach is better and why?
Question 2:
I decide to sequentially launch 10 new changes in a product. I run 10 A/B t-tests in sequence one after other for lift of 5 units each. Assuming, each of the changes gave me a significant lift, I believe in the end I will get a total lift of 50 units. Is this accurate? If not, what things should I be considering here or what more pieces of information do I need?
Last edited: