Calculate required sample size

[richardjack]

Hello,

I have a test for the thermal performance of a building with a defined uncertainty margin (15%). I want to use it to measure the effects of refurbishments designed to improve thermal performance, but the effect of the refurbishments can be quite small (5-50%). In order to have confidence in the effect of a particular type of refurbishment, we measure across a sample of buildings. I'd like to calculate the minimum required sample size to have a particular confidence (say 95%) in the measured effect of refurbishments for a range of anticipated improvements (e.g. increasing in 5% increments from 5%).

Do I have enough information to reliably calculate a suitable sample size? And if so, what is the process to do so?

Thanks!
Richard

 

[noetsi]

these days this is commonly done in the context of statistical power. You decide what power you need and determine a sample size based on that. What statistical method are you using?

 

[richardjack]

Thanks! Sorry to be dumb, but I'm not 100% sure what's meant by power in this context... would it be suitable to say I was looking for a 95% confidence level? For the statistical method I'd appreciate some guidance, my ill-informed Googling suggests that something like a paired samples t-test might be suitable?

 

[Miner]

95% confidence refers to Type 1 error (alpha risk), which is controlled by the alpha threshold you use to determine whether a p-value is statistically significant (e.g., 0.05 in your case). Power is another way of expressing the Type 2 error (beta risk), which is controlled by the sample size. Power = 1 - beta.

Type 1 error is the risk that you reject the null hypothesis when the null hypothesis is true (i.e., false positive).
Type 2 error is the risk that you fail to reject the null hypothesis when the alternate hypothesis is true (i.e., false negative).

 

[richardjack]

Thanks! So one would choose a power level they were confident in, as they would a confidence level. Having done so, how would I use the known uncertainty level of my measurement method to calculate suitable sample sizes to measure different sizes of difference between pre- and post-refurbishment performance (I think this could be described as being confident in a hypothesis that the performance in the two samples was different)?

 

[Miner]

This is a link to a sample size calculator for comparing 2 independent means.

 

[noetsi]

There are a lot of free power tools. You might look up gpower. https://stats.idre.ucla.edu/other/gpower/

 

[richardjack]

Thanks! These are really helpful. One more question... is there a version of the calculation I can carry out without estimating the standard deviation? It's hard to know what that would be in advance, and can very quite widely depending on the house types measured. If that's not possible, perhaps I could do a sensitivity analysis with a few different standard deviation estimates.

 

[Miner]

Thanks! These are really helpful. One more question... is there a version of the calculation I can carry out without estimating the standard deviation? It's hard to know what that would be in advance, and can very quite widely depending on the house types measured.

Short answer is No.

If that's not possible, perhaps I could do a sensitivity analysis with a few different standard deviation estimates.

Excellent idea.