95% CI is Insignificant, P-Value is Significant

#1
Suppose you're performing a 2x2 contingency table analysis. You do your Chi Square or Fisher Exact test, as appropriate, then you calculate an odds ratio and get your 95% CI for the OR. Under what circumstances can the P-Value for the ChiSquare and Fisher test be significant (e.g. P < 0.05) but the 95% CI of the OR straddles 1.0?

Is this ever possible using standard statistics techniques?

What about if you're performing complex survey analysis where the survey is stratified, clustered and weighted? I ask because I've been using SAS to process data from a complex survey design and I've run into a situation where the P-Value and 95% CI conflict:

Sample Size = 3,618

Rao-Scott Chi-Square Test
Pearson Chi-Square 4.1614
Design Correction 1.0794
Rao-Scott Chi-Square 3.8551
DF 1
Pr > ChiSq 0.0496 *** significant
F Value 3.8551
Num DF 1
Den DF 1765
Pr > F 0.0498 *** significant


Odds Ratio (Row1/Row2)
Estimate 95% Confidence Limits
Odds Ratio 0.1548 (0.0176-1.3595) *** not significant
 

fed1

TS Contributor
#2
Yes this is possible.

It occurs because the chi square (which is just the square of the asymptotic z-test for proportions) cannot be inverted to give the confidence interval for OR that you are using.

In general the deal with p-values and CI that you are thinking of only holds if the hypotheis test and CI are connected by a tautoulogy, excuse spelling.

IE

CI {estimate -se p zquant < parameter < estimate + se* zquant}

cang be reexpressed with the test stat in the middle of the inequality

I would be interested to see exact conditions myself, but that is going to be tedious i think
 

Dason

Ambassador to the humans
#3
Essentially what fed1 is saying is that the program is using one method to compute the p-value and it's using a different method to compute the confidence interval and there is no guarantee those methods will match up exactly. Asymptotically if they're both good methods they should give similar results but we can't always invert a hypothesis test to get a confidence interval very easily but there might be an easier method that works asymptotically...