Power analysis and chi-squared control group sizes.


Quite new to stats, I've asked one question on here before and received a lot of really helpful information. If I can be helped again I'd be really grateful.

at my job, the senior management, want to have the smallest control group possible, in order to expose the maximum amount of people to the treatment. The treatment in this case is an email or a telephone call, some sort of communication. The positive outcome, or goal is an investment in our product.

I've looked into Power analysis and mocked up scenarios, testing different control percentage splits such as 50/50, 70/30 etc.

I was asked how small could we make the control group with a large proposed communication to 300,000 people.

I will use chi-squared to compare control and test. the comparison is between those that bought our product and those that didn't. I appreciate with such a large sample size, it's easy to get a miniscule p-value.

If the investment rate in control is 5% and the investment rate in the test group is 6%, this will cause a tiny p-value. If the effect size (phi) is calculated, it indicates a very tiny effect. However, I'm not concerned that the effect is really small, I'm leaving it up to senior management, to decide whether a 6% rate of investment is worth the trouble of the communication, compared to 5% in control.

I mocked up some chi squares to look at the 5% investment rate vs 6% and played with different control splits (based on overall sample size of 300,000.

I conclude that a control group of just 5% (equating to 15,000 people) would be fine. I used power analysis. I plugged in the tiny effect size (w=0.009213167) , the sample size (300,000) the sig level( 0.05) and power comes out at 0.99. This is above the desired 0.8 level. So I believe a 5% control is ok in this instance, as the sample size is so large.

If I change the investment rates to 6% (test) vs 5.5% (control) with the same sample size and 5% (15,000 people) control split, this is still statistically significant, however, the already tiny effect size gets even smaller, which means power drops to 0.71. Therefore I will report that we cannot detect effects as small as this, using this 5% control split. (Because power drops below 0.8)

I tried other splits @ sample = 300,000.

For example: a 1% control cell. So that's 3,000 in control group and 297,000 in test.

as above, if 6% invest in test and 5% in control, using chi squared, this would be a p-value of 0.02. So this is significant.

effect size is 0.004 (tiny but not concerned about this)

power then equals 0.63. Less than the 0.8 figure.

So this leads me to believe that a 1% control cell for 300,000 sample size, with this difference in reinvestment (5% vs 6%) (effect size) is not useable. The power is less than 0.8.

My question to you: Am i interpreting this correctly? If I ever get a statistically significant difference and reject the null hypothesis, should I check the power? If the result is significant, but the power is less than 0.8, does this mean that I should not reject the null hypothesis?

I've been asked to draft what control splits we should use for sample sizes starting from 10,000 and going up in 10,000 increments. I planned to use the 5% and 6% investment example and use as smaller control as possible. Their goal is to communicate to as many people in 'test' as possible.

I made a start:

for sample size of 10,000: using same 5% and 6% investment (control and test), using a 30% control (3,000 people, 7,000 people in test) this causes a p-value of 0.04. Significant. However, power is 0.5

THEREFORE, do I reject the null hypothesis or not? Result is significant, but power < 0.8.

Is "power >= 0.8" essential, therefore I should choose a control size that achieves this?

Thank you so much.

Sorry for the essay.

Dr Strangelove