Power

[danony]

Hi there,

I need some help getting my head around this power concept.

If there's a small but important effect, should you try and find it however you have to do that? I.e. if RT is associated with death for example then it seems it's important enough to find it. For some reason I think this is wrong though. Is it that I'm confusing research design (where you need to maximise the chances of finding an effect) with statistical significance tests (conservative with regard to the null)?

EDITED

[Dason]

Please only post a topic once. I'm going to delete your other thread as it seems more appropriate here. Thank you. - Dason

[danony]

sorry about that - thanks.

[trinker]

You dilemma has no answer. I'm sorry to tell you this but many others with great minds, including Tukey, have wrestled with this issue. If you have enough power you will find a significant difference (probably). You then report this along with CI's and strength of effect measures. You let the data and the results speak for itself. People will read your work in that field and make decisions based on your evidence.

It was a sad day when I realized stats doesn't prove anything. It just provides evidence. You are providing evidence and building a case. it is up to the literature consumer to make an informed judgment about your work based on what you've provided (or not provided).

Hope this is helpful. Sorry if it's not the clear cut answer you were looking for.

[danony]

Thanks trinker. I take your point on the language. I have a lot of ASD tendencies - mainly in being incredibly literal/concrete - so I use the word a lot in relation to myself! I certainly didn't mean to be offensive. I've edited it out now.

Well at least you didn't point out a massive flaw in my thinking in regard to the power equation. Hopefully that means I'm at least on the right track with that bit of it.

[noetsi]

Usually you don't discuss the limits such as a lack of power on your research in the introduction, unless you are correcting (or think you are) a past problem. In that case you would mention what you did different and why you think it matters.

So if you are measuring something in a different way and think the effect will be more likely to be detected (and/or more likely to be statistically significant) as a result, then you should point this out right after your literature review. It helps a lot if you can establish that your method is better (or some have suggested applying your approach) in the literature review.

[danony]

Thanks very much noetsi - I am trying to correct a past problem. This has been really helpful - also to know that I should mention why my approach is 'better' after the lit review. I'm focusing most of my lit review of the RT measure to highlighting differences between novices and experienced so I guess pointing out the power problem after that, rather than before, will make it easier to for the reader to understand.

[noetsi]

You are welcome. My comments were driven in part by being rapped for discussing limits on my study (as compared to previous studies by others) before my findings in a recent research class. That still stings... You point out weaknesses in others in the lit review (and maybe introduction and purpose section) and limits on your study in the findings or limitations section.

The reason to discuss how your approach works better after the lit review (there is no formal rule of course) is that you can point out literature which shows there was problems with the alternative you are not using and (hopefully) research that supports your approach first. Then you have a better case for your argument.

Academics particularly look for that (real world types may or may not, but I would guess this is for academics if you are doing a lit review). :)

[Jake]

For some reason I think is 'wrong' - but surely the whole point of an experiment is to find an effect if it's there (but I keep thinking...no! I am supposed to be conservative). But surely if it's a small but important effect, you should try and find it however you have to do that. I.e. if RT is associated with crashing (death) then it's important enough to find it.It's not wrong at all IMO. It's smart research design. I think your reasoning here is pretty much spot on.


Is it that I'm confusing research design (where you need to maximise the chances of finding an effect) with statistical significance tests (conservative with regard to the null?)?I think what you're confusing are the issues of getting a biased parameter estimate versus getting a biased effect size for that parameter estimate (where "bias" in this context simply means "systematically different from what we would find if we were to exhaustively sample the entire population of interestion").

If the "true" relationship in the population is linear, then this extreme sampling strategy will not result in a biased parameter estimate. However, if the true relationship is nonlinear, say quadratic, then the parameter estimates from this strategy can sometimes be a bit misleading (although I'm not sure I'd call them "biased"). It takes at least 3 parameters to define a quadratic function -- an intercept, a slope, and a curvature parameter, even if some of these are implicitly 0 as in the case of y = x^2 = 0 + 0*x + 1*x^2 -- so your predictor needs to have at least 3 unique values of x if you are to estimate a quadratic relationship. Your predictor will only have 2 unique values, "novice" and "expert."

The estimated effect size here will be biased, in the sense that I outlined above. But you are probably not interested in obtaining an unbiased estimate of the "true" effect size in the population anyway. You are simply interested in demonstrating that, whatever the true effect size is, it is non-zero.

[danony]

Hey Jake,
You have hit another one of my confused bits of grey matter right on the head. I've been so stuck on this particular point I'm incredibly relieved you addressed it. Thanks for that. I don't have any reason to believe there is a non linear relationship between experience and RT. I think I'm in love with this forum. <3

Thanks everyone. This is really helping me get my head around the details of what I can and can't argue about my research design and the results it comes up with (probably nothing lol since it's an honours thesis). I came up with this design in 5 seconds based on previous research. It was approved in 5 seconds by my supervisor too. The strange thing is I had no idea why it was the right design for the job so I've been scrabbling around trying to unpick the assumptions behind it ever since.

I feel like singing from a mountaintop now that this is cleared up. ahem. YODELLAAEEEOOOO!