Assessing sample-size after data collection

[felixg]

Hi all! I ran a correlation study with the maximum sample size that was possible, and found some strong significant effects as well as non-signifiant effects. I'm now trying to ascertain whether the sample size was appropriate.

I ran a post-hoc power analysis, and found output parameters ranging from .59 for the variables with the lowest correlation coefficients that were still statistically significant, to .9 for the highest coefficients. I'm inferring from this that the study was under-powered, but I've been reading that post-hoc power analysis is not recommended.

Any other ideas as to how I can check whether the sample size was appropriate, retrospectively?

Many thanks!

 

[ondansetron]

Hi all! I ran a correlation study with the maximum sample size that was possible, and found some strong significant effects as well as non-signifiant effects. I'm now trying to ascertain whether the sample size was appropriate.

I ran a post-hoc power analysis, and found output parameters ranging from .59 for the variables with the lowest correlation coefficients that were still statistically significant, to .9 for the highest coefficients. I'm inferring from this that the study was under-powered, but I've been reading that post-hoc power analysis is not recommended.

Any other ideas as to how I can check whether the sample size was appropriate, retrospectively?

Many thanks!

Post-hoc power analysis doesn't provide much value (or lend for interpretation) to the current study. Power and sample size calculations are meant to be done before hand. Further, it's readily demonstrable that post-hoc power calculations are transformations of the given p-value for the specified test.

The best you can do retrospectively, I believe, is to use your judgement. If a test is significant, then by definition it cannot be under powered. However, there is the possibility of gross overestimation of effect sizes more than if the study was appropriately powered (due to more sampling variation). If a test did not meet significance, you could mention that a possible reason is under-powering, but there are other reasons that may be at play.

Other things to consider regarding sufficient sample size are whether you had categories with very few or no observations-- could mean the sample size was too small for the phenomenon you were studying.

 

[hlsmith]

I second ondansetron's comment. I had a similar scenario recently and when looking at the literature, post hoc power calculations become conditional on what you already did and can be uninformative. This could also possibly fall under "investigator degrees of freedom".

 

[felixg]

Thanks very much to both of you! I am writing up an article on this study and I'm required to explain if I believe the sample size is appropriate. If there are any other ideas on how I can assess that in some kind of objective or report-able way, I would be most grateful.

Thanks again!