Help - is a collection of studies normally distributed if the n >150? false positives

#1
Hi,

I have a collection of medical trials, all for different conditions and different medications. There are 160 in all. They have all different methodologies etc. They come up with a result of either "positive" drug worked or "negative" no effect of drug.

I want to estimate the number of false positives.....

Can I say that the number of false positives is 1/20 (5%) - by assuming the trials are normally distributed as n is large?

Thx

Liz
 

Karabiner

TS Contributor
#2
Re: Help - is a collection of studies normally distributed if the n >150? false posit

No variable becomes normally distributed just because sample size is large.

You could asses the probability of a false rejection in each trial, using
its actual p-value, and combine these probabilities across trial. But
that's just theory. Here you can read what really is the case.

With kind regards

K.
 
#3
Re: Help - is a collection of studies normally distributed if the n >150? false posit

Many Thanks K!!!

So, can I just confirm - for this meta-analysis of 160 different trials testing the efficacy of a particular "style" of treatment - its wrong to say 8/160 will be "positive" due to chance alone - the only way to assess probability of their being "false positives" is to analyse this in each trial and somehow gather together the results (apologies for repeating myself, I find this a bit tricky)

Many Thanks

:) :) :)
 

Karabiner

TS Contributor
#4
Re: Help - is a collection of studies normally distributed if the n >150? false posit

I a meta-analysis, one doesn't count significant
results and then calculates how many of them
are false. Admittedly, I do not know what such
an analysis could be good for.

In a meta-analysis, one tries to synthesize the
statistical information (effects sizes measures)
from single studies, which are based on samples,
in order to estimate the effect in the underlying
population. So I am not sure if you really want
to perform a meta-analysis, or perhaps something
different.

With kind regards

K.
 

EdGr

New Member
#5
Re: Help - is a collection of studies normally distributed if the n >150? false posit

If Sustat has some number of trials, regardless of the nature of the analyses or the sample size, if each test is done at alpha level 0.05, we would indeed expect 5% significant by chance if all null hypotheses were true. I think you could compare the observed percentage to that. Since these are all different topics, there is no basis for combining them. I assume you want some very general idea what percentage of studies of this sort have findings beyond chance. Say 20% of them are significant at 0.05 or less. I think it is legit to subtract 5% and say about 15% have non-type I error significance. This kind of thing is often done with genetics studies with thousands of analyses. Look up "False Discovery Rate" to learn more.

Just remember that small studies that are not significant often don't get published (file drawer bias). But you say these are all large N. Good.