I’d like comments concerning our statistical plan in a science fair project. The project tests device X against a control device. Using either device has a binary outcome of success or failure.

First, we performed 100 trials with each device and found 66% success with X and 61% success with control. We used a Power table for Cohen’s Effect Size d = 0.2, two-tailed alpha = 0.05, and power of .80 to decide that we needed at least 393 trials with each device. We’ll try to do more trials, but hopefully we’re in the ballpark.

Now, we don’t have the trials done yet but our next calculation would be an independent two-sample t-test for equal sample sizes, equal variance after http://en.wikipedia.org/wiki/Student's_t-test.

Finally, with our t statistic and degrees of freedom, we’ll use a t-table to find a p-value.

(I tried this post on another statistics site with no reply, so any expertise here is greatly appreciated.)