Confidence Interval in Statistics test

[jmvk]

Hi all,

I just had a test with the following question, of which I doubt the 'correct' answer:

Taking a level of significance of 5% in a hypothesis test and not rejecting the null hypothesis, is the same as saying:
A. We are 95% confident that the results have occurred by chance
B. We are 95% confident that the results have not occurred by chance
C. We are 5% confident that the results have not occurred by chance
D. None of the above

According to the answers, B is correct. How is this true? For example, if the p-value is 10%, how can we be 95% confident that the results have not occurred by chance?

 

[hlsmith]

D - they are all incorrect since you can't say we are ## confident. The convoluted definition is closer to, given repeated sampling of the population and correct model specification, 95% of confidence intervals will contain the true estimate. I think I got that wording right, but I am sure someone will correct me on nuances.

 

[jmvk]

D - they are all incorrect since you can't say we are ## confident. The convoluted definition is closer to, given repeated sampling of the population and correct model specification, 95% of confidence intervals will contain the true estimate. I think I got that wording right, but I am sure someone will correct me on nuances.

Thanks a lot!

 

[ondansetron]

D - they are all incorrect since you can't say we are ## confident. The convoluted definition is closer to, given repeated sampling of the population and correct model specification, 95% of confidence intervals will contain the true estimate. I think I got that wording right, but I am sure someone will correct me on nuances.

So, in the traditional sense of frequentist statistics, I think you can say "95% confident" because they didn't mean it as a probability statement on the interval. They meant it to refer to the methodology and long run success rate if used properly. Almost a short hand was of saying "this interval was generated by a method where 95% of all possible intervals contain the true parameter value and 5% don't, and this interval is X,Y". In other words, confidence has a different meaning than "probability" or "surety."

This discussion is aside from the question posted, because I think the question posted is a bad question.

 

[hlsmith]

If I was absolutely pressured to select between A-C, I would pick B. What they are probably trying to get at is that given alpha = 0.05, we are willing to accept results as extreme as revealed by chance 5% of the time, thus the opposite of that vague interpretation would be we are 95% sure the results are not by chance.

I think another component is whether this is for say a high school test or graduate level examine, not that it should ever matter, but the vague interpretation may help to deliver the general idea in a basic course, but the real concepts of significance testing based on p-values and confidence intervals is vary nuanced.

 

[Bruce Wilson]

Hi, I'm new here. I have a related question concening confidence intervals and statistical significance.

I have always assumed that a confidence interval that crosses the null value (1 in the case of OR) signifies non-statistical significance. However, I came across a paper on odds ratios by Magdalena Szumilas that states, the following:

"It is important to note however, that unlike the p value, the 95% CI does not report a measure’s statistical significance. In practice, the 95% CI is often used as a proxy for the presence of statistical significance if it does not overlap the null value (e.g. OR=1). Nevertheless, it would be inappropriate to interpret an OR with 95% CI that spans the null value as indicating evidence for lack of association between the exposure and outcome.”

This seems to belie everything I’ve read about CIs. Also, it is possile to calculate a P-value from a CI. What do you make of this. Is she wrong?

The paper is here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2938757/ Her statement is under the section: "What about confidence intervals?"

Any clarification on this would be greatly appreciated.

Bruce Wilson

 

[hlsmith]

Bruce Wilson next time open your own thread and don't piggyback on an existing one, since your question is a little different than the OP's.

Yeah, ignore her on this, since her description is not straightforward. I am guessing she is trying to say p-values are used for decisions about significance while ORs are association measures and using fast and hard rules based on CIs crossing 1 may negate the informative properties of the OR. Which is true in some regards, and I don't think she is trying to say they don't align with pvalues, because they do. You are right in your background knowledge. However, pvalues are frowned upon these days since there is no gray area, such as a p = 0.051 may not be considered of interest. This same logic holds for CIs, which may be what she is trying to get at.

Give me a second I am gonna read the article in its entirety.

 

[hlsmith]

I skimmed the article, I think that line is just a little unclear, especially since she writes the following:

"
This example illustrates a few important points. First, presence of a positive OR for an outcome given a particular exposure does

not necessarily indicate that this association is statistically significant. One must consider the confidence intervals and p value

(where provided) to determine significance."

So this text aligns with your background knowledge.

 

[GretaGarbo]

I have always assumed that a confidence interval that crosses the null value (1 in the case of OR) signifies non-statistical significance.

I would say the same.

I believe that the author of that text is just wrong. (Of course, that said, I could be wrong. LOL)

There is a correspondence theorem. That a significance test "corresponds" to a confidence interval. Actually, one usual way to construct a confidence interval is to do repeated significance tests. For all those mu:s (hypothetical expected values) that you can not reject with a significance test will be within the confidence interval, and those values of mu that you can reject will be outside of the confidence interval.

That correspondence theorem is in the Casella-Berger (a statistical text book that is not on the elementary level).

There is nothing special with the odds ratio. The odds ratio is just exp(beta), where beta is the estimated parameter in the logistic regression. If beta under the null hypothesis is zero (beta=0) then exp(0) =1. So the odds ratio of 1 just corresponds to a beta of 0. And that is like the usual hypothesis testing.

Conclusion: A confidence interval corresponds to a significance test.

 

[Bruce Wilson]

Thanks for your reponses, folks. Sorry for piggybacking. I'm new here.

 

[hlsmith]

No biggie Bruce. Feel free to post as many questions as your heart desires in new threads. We look for to them and your contributions. Welcome aboard.

 

[j58]

I have always assumed that a confidence interval that crosses the null value (1 in the case of OR) signifies non-statistical significance. However, I came across a paper on odds ratios by Magdalena Szumilas that states, the following:

"It is important to note however, that unlike the p value, the 95% CI does not report a measure’s statistical significance. In practice, the 95% CI is often used as a proxy for the presence of statistical significance if it does not overlap the null value (e.g. OR=1). Nevertheless, it would be inappropriate to interpret an OR with 95% CI that spans the null value as indicating evidence for lack of association between the exposure and outcome.”

This seems to belie everything I’ve read about CIs. Also, it is possile to calculate a P-value from a CI. What do you make of this. Is she wrong?

The paper is here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2938757/ Her statement is under the section: "What about confidence intervals?"

The author's last sentence is correct, but the point is subtle and should have been explained. The 2-tailed p-value for the OR will be greater than .05 exactly when the 95% CI for the OR includes 1 (assuming both are calculated in the usual way). But it is incorrect to interpret a non-signficant p-value as lack of association, because that is equivalent to accepting the null hypothesis, when the best we can say is that we have failed to reject it. For exactly the same reason, it is incorrect to interpret a CI that includes the null value as indicating lack of association.

 

[Bruce Wilson]

Thanks j58! That clarifies things.