Statistical plan in science fair


I’d like comments concerning our statistical plan in a science fair project. The project tests device X against a control device. Using either device has a binary outcome of success or failure.

First, we performed 100 trials with each device and found 66% success with X and 61% success with control. We used a Power table for Cohen’s Effect Size d = 0.2, two-tailed alpha = 0.05, and power of .80 to decide that we needed at least 393 trials with each device. We’ll try to do more trials, but hopefully we’re in the ballpark.

Now, we don’t have the trials done yet but our next calculation would be an independent two-sample t-test for equal sample sizes, equal variance after's_t-test.

Finally, with our t statistic and degrees of freedom, we’ll use a t-table to find a p-value.

(I tried this post on another statistics site with no reply, so any expertise here is greatly appreciated.)
Hi Trinker,

T-test seemed appropriate and easy enough, but we're not locked in. What would you suggest? If we stick with the t-test though, how do you like the rest of our plan?

Thanks for asking.


You're violating assumptions by using the t-test, one of which is that your data is drawn from a random population (You check this by looking at the residuals). Your distribution will likely binomial. Have a link at this LINK.