1. G*Power Sample Size Calculation

Hi Everyone,

First, let me say that I am thrilled to find this forum! It has been a while since I have taken Stats I and II, and I will be the first to admit that I struggled my way through it then. I have a project that is going to require some data analysis coming up, and I was wondering if anyone would be willing to look over my sample size calculation and tell me if I am on the right track? Here's what I have:

Design: 2x3x2 Three-way factorial ANOVA
Population: Unknown (est. is greater than 100,000). Single group.
Confidence Interval: 95%
Effect Size: Small or Medium

I am using G*Power 3.1.3, and the information below is what I have entered in the fields. I have colored the areas that I am really not certain about, but if you see anything else that looks out of place, please let me know.

Test Family: F Tests
Statistical Test: ANOVA, Fixed effects, special, main effects, and interactions
Type of Power Analysis: A priori

Effect size: .10
error prob: .05
Power (1-b error prob): .8
Degrees Freedom: 2
Number of groups: 12

Thanks,

2. Re: G*Power Sample Size Calculation

Are you doing a fixed effect ANOVA or is your analysis using random effects? I did not see that in your comments (although most use fixed effects). Remember if you are using a priori this is supposed to occur before you have seen the data (a point I consider artificial but the field does not). Is it reasonable that the effect size is .1 (I don't know why you decided on this value).

4. Re: G*Power Sample Size Calculation

Hi,

Thanks for the response

This is social science research (survey method) and I am using the variables of introvert/extrovert (2 levels), motivation category (3 levels), and gender (2 levels) on time. Thus, I would think they would all fall in the category of "fixed."

I have not seen the data yet. I know I will need to do a post test once it is collected. I chose .1 for the effect size because I thought that a smaller effect was better, but I need to calculate the sample size correctly before I can see if that will even be feasible. If its not, then I am most likely looking at a .25 for a medium effect, correct? I think that medium is the standard. I also noticed that the power defaults to .95, I wonder if I should use that rather than a .8? Any thoughts?

5. Re: G*Power Sample Size Calculation

Is not personality and motivation typically measured using continuous scales? If you have taken a continuous scale and made it into a categorical one this can have a range of negative implications for your analysis.

7. Re: G*Power Sample Size Calculation

Originally Posted by Lazar
Is not personality and motivation typically measured using continuous scales? If you have taken a continuous scale and made it into a categorical one this can have a range of negative implications for your analysis.
In this particular survey, the personality variable will be measured using 1 self-report item. As far as the motivation piece, the empirical tool that I am using defines a primary motivation (there are 3 types) based upon a point system. I feel that this is best described by the author of the measure. According to Yee (2007), "Because standardized scores and effect sizes (based on continuous variables) are less-interpretable than percentages (based on categorical variables), a different way of understanding this data is presented here. The “primary motivation” for each player was inferred from their scores. A respondent was assigned a primary motivation if there was no close secondary motivation (primary * .75 > secondary). 57% of players were assigned a primary motivation based on this criteria. This is a somewhat lax criteria but serves the purpose of providing an easier interpretation of the data".

If I may ask, what types of implications do you think that I could be looking at here?

8. Re: G*Power Sample Size Calculation

Loss of power and bias and the two main ones. Is the personality item from one of the 5 item big-five measures? Is so, is it not on a five point scale?

EDIT:

This puts it nicely

The fact that some people murder doesn't mean we should copy them. And murdering data, though not as serious, should
also be avoided.
-- Frank E. Harrell (answering a question on categorization of continuous variables in survival modelling)
R-help (July 2005)

9. Re: G*Power Sample Size Calculation

No, it is a self-classification based on a description from existing studies. There is no scale on that item, its either introvert or the extrovert. How does the continuous vs. categorical data impact sample size?

10. Re: G*Power Sample Size Calculation

Not sure I know what you mean by "how does it impact sample size". Do you mean how does it impact power?

11. Re: G*Power Sample Size Calculation

Yep, that is what I am wondering. Would I need to make additional adjustments to my sample size calculation to increase the power somehow?

Here is what I am looking at right now as my preliminary sample size calculation:

Effect size: .25 (I think you are correct that .1 may not be feasible)
error prob: .05 (Standard)
Power (1-b error prob): .95 (The higher the power, the better right?)
Degrees Freedom: 2 (calculated by (2-1) * (3-1)* (2-1) =2)
Number of groups: 12 (calculated by 2x3x2)

I put this into G*Power and came up with: Total sample size 251

12. Re: G*Power Sample Size Calculation

This is social science research (survey method) and I am using the variables of introvert/extrovert (2 levels), motivation category (3 levels), and gender (2 levels) on time. Thus, I would think they would all fall in the category of "fixed."
Fixed means, commonly, that you are using all possible levels of a variable and that you are only interested in the specific levels you measure. Random effects involve sampling a group of levels from a wider subset of effects - when what you are really interested in is the larger subset not just what you sampled. I would guess you are doing fixed from your description.

I have not seen the data yet. I know I will need to do a post test once it is collected. I chose .1 for the effect size because I thought that a smaller effect was better, but I need to calculate the sample size correctly before I can see if that will even be feasible. If its not, then I am most likely looking at a .25 for a medium effect, correct? I think that medium is the standard. I also noticed that the power defaults to .95, I wonder if I should use that rather than a .8? Any thoughts?
A priori has more power, you don't have to do post hoc test if you do a apriori calculation (although commonly this is ignored). You should base your effect size on what you think actually exists in the population. If you don't know that using a conservative number is your best alternative. Is there any literature such as a meta analysis or a collegue's research to suggest what the effect size is?

Each specialization has its own rules for what a small, medium, and large effect size is. Cohen's suggestions are commonly utilized.

13. Re: G*Power Sample Size Calculation

Originally Posted by noetsi
A priori has more power, you don't have to do post hoc test if you do a apriori calculation (although commonly this is ignored).
That is good to know!

I'm a little boggled by the effect sizes. I have not been able to locate a meta-analysis, so I am going to have to go with Cohen's d. However, my textbook says .2 is small, .5 is medium, and .8 is large. G*Power says .1 is small, .25 is medium, .5 is large. When I google Cohen's d, I get a table that says .1 is small, .30 is medium, and .5 is large. Now that is just plain confusing lol. Is this what you meant by differences in specialization?

14. Re: G*Power Sample Size Calculation

Is this what you meant by differences in specialization?
the problem with effect sizes is that what Cohen intended to act as a general guideline and a mere set of suggestions became dogma to many people and now it gets thrown around as if it were the word of God ( much like the whole p < .05 thing). for instance, in areas like mine where a lot of research is correlational, we're quite happy with an effect size of 0.5. if we find something around the 0.8 range we even get suspicious because it's simply too high... or we ignore it. it's important that one knows his/her area enough to know which effect sizes are in which range... or, to be honest, if at some point this becomes too much of a hassle just take the Cohen (1988) citation and run with it. it's such a well-known reference and so many people use it that you just can't be all that wrong by using it as well...

15. Re: G*Power Sample Size Calculation

Originally Posted by spunky
it's important that one knows his/her area enough to know which effect sizes are in which range... or, to be honest, if at some point this becomes too much of a hassle just take the Cohen (1988) citation and run with it. it's such a well-known reference and so many people use it that you just can't be all that wrong by using it as well...
That sounds like a good idea! Since G*power seems to be the most conservative estimate, I am going to compare that with Cohen (1988) and see how it stacks up. Thanks!

*Update* I looked it up and what I found when I googled (Wikipedia) was identical to what Cohen (1988) reports. I'm not a fan of Wiki at all, but perhaps it is useful sometimes!

16. Re: G*Power Sample Size Calculation

I'm a little boggled by the effect sizes. I have not been able to locate a meta-analysis, so I am going to have to go with Cohen's d. However, my textbook says .2 is small, .5 is medium, and .8 is large. G*Power says .1 is small, .25 is medium, .5 is large.
That's not for d, I suppose. Perhaps for f?
When I google Cohen's d, I get a table that says .1 is small, .30 is medium, and .5 is large.
Definitely not. That's for r, not for d.

Kind regards

K.

17. Re: G*Power Sample Size Calculation

Originally Posted by Karabiner
That's not for d, I suppose. Perhaps for f?

Definitely not. That's for r, not for d.

Kind regards

K.
And of course it depends on the discipline you're in as well...