Adjusting confidence intervals for multiple comparisons

[lancearmstrong1313]

Hi all, someone asked me a question and I wasn't quite sure how to BEST go about solving it in SAS. It involves creating confidence intervals but adjusting them for multiple comparisons.


Let's say I don't actually have any raw data, just summary stuff like means, sample size, variation.... and I want to compare a number of groups.

I would want to create confidence intervals comparing the differences between the possible pairwise combinations (A-B, A-C, A-D.... B-C....) and then adjust them accordingly (such as using a multiplier of 1-.05/3 for 3 groups for a 95% CI for Bonferroni). This would be quite easy if I had the actual data - I could use PROC GLM for instance. However, since I do not, is there any easy way to compute adjusted confidence intervals in SAS?

I know PROC MULTTEST adjusts p-values but I could not find anything that would do the same for confidence intervals. Other than writing your own code, does SAS have any type of built in procedures that you could input the means, variation, sample size...and it would compute the adjusted confidence intervals for each possible pairwise comparison?

[hlsmith]

Not familiar with any built in procedure. Code could definitely be written. Interested to see what anybody says. If you want to try to write the code I would be happy to examine this topic as well (would probably have to use bonferroni or one of the simple corrections).

[victorxstc]

I onced asked a similar question and Karabiner replied me, but I can't remember the location (and trying to find it but to no avail).

But like the Bonferroni method which does something clear with the alpha, some clear calculations are done at the CI level in order to apply the correction of multiple comparison problem. So the only thing you have to do is that instead of calculating 95% CIs, you have to tell SAS to calculate for example 90% or 92% or 86% CI. The more the number of multiple pairwise comparisons, the lower the level of confidence.

How this 90% or 87% or other confidence levels are determined for correcting the MC problem? I don't know, but think that is search-able, and experts here know it.

[Dason]

It's actually the other way around - the more intervals you do you use a higher confidence level. Intuitively we would want our confidence intervals to get wider as we increase the number of intervals we make. This happens when we increase the confidence level.

And it looks like lancearmstrong already mentioned how you actually do the adjustment in the case of bonferonni. Instead of using (1-.05)% confidence level you use (1-.05/k)% where k is the number of intervals you make.

[lancearmstrong1313]

Yes, the intervals would get wider as more comparisons are done.

[lancearmstrong1313]

@victorsxtc - if you go to your profile and search under your started threads, maybe you will have a better shot at finding the thread with the code you mentioned?

[victorxstc]

I swear I am searching for half an hour right now, or more I think. But I can't find it. All I need is to manage to tell the search feature to search for:

victorxstc AND "multiple comparison" AND "confidence interval"... but the search feature can't understand this... It sorts me hundreds of threads with ANY of the following keywords:

victorxstc
multiple
comparison
and
confidence
interval



but I checked my subscribed threads, and all, but couldn't find it... I will update later because my brain is burning right now! But can remember Karabiner told me that the confidence limit decreases when we want to correct the MC problem. Would be glad to learn it through our conversation here.

[victorxstc]

Hey your hint worked perfectly you are a life saver

the link is here, I had remembered quite incorrectly

http://www.talkstats.com/showthread....ere?highlight=

Aha, yes I understand now We "have" to find a more accurate confidence interval to compensate for the increased chance of random significance

What I had memorized from that conversation was that "our confidence level is decreased by MC"... but had forgotten this part: [So if we calculate 95% CI, we are really calculating 92% (for example) CI. So we must calculate 97.5% CI in order to obtain the 95% CI at non-MC condition]

That was what I remembered and incorrectly told in a way that became quite reverse

[hlsmith]

Yes, this question came up during my comprehensive exam, and I starting explaining it the wrong way at first. Then figured it out while they were convening (irk me something fierce). I would say that the code would need to have an output data set that pulled the variables of interest then a couple of statements in a datastep. If you want to grab a dataset from SAS.help I would be eager and happy to try and help write the code.

[lancearmstrong1313]

I've been messing around with it a little bit and just created a made up data set like this:

data test;
input mean sd n;
cards;
10 2.5 40
15 2.0 37
08 2.3 45
17 3.4 35
11 2.9 41
;
run;

So I would want to find an interval for the difference between A-B (row 1 - row 2) and so forth for every pairwise combo;

[hlsmith]

So in the hypothetical we do not have the actual data only means, standard deviations, and n. Say we are reviewing a journal as in a meta-analysis. Correct? Now you want to calculate the differences and respective corrected 95% CI?

working on some code.

[hlsmith]

Probably going to checkout soon, but look at these two links:

https://onlinecourses.science.psu.edu/stat505/node/128

http://sites.stat.psu.edu/~ajw13/sta...ata/swiss3.txt

[hlsmith]

This isn't quite it yet but check out below - we still need to figure how to enter all of the factorial combinations, here I just had the first variable against the other four, but obviously that is incorrect for all of the pairwise combos. Will play with it tomorrow unless you or someone else gets it first.

data test1;
input Variable n1 xbar1 s21;
datalines;
1 40 10 2.5
2 40 10 2.5
3 40 10 2.5
4 40 10 2.5
;
run;
data test2;
input Variable n2 xbar2 s22;
datalines;
1 37 15 2.0
2 45 08 2.3
3 35 17 3.4
4 41 11 2.9
;
run;
data combineTest;
merge test1 test2;
by variable;
f=finv(0.95,&p,n1+n2-&p-1);
t=tinv(1-0.025/&p,n1+n2-2);
sp=((n1-1)*s21+(n2-1)*s22)/(n1+n2-2);
losim=xbar1-xbar2-sqrt(&p*(n1+n2-2)*f*(1/n1+1/n2)*sp/(n1+n2-&p-1));
upsim=xbar1-xbar2+sqrt(&p*(n1+n2-2)*f*(1/n1+1/n2)*sp/(n1+n2-&p-1));
lobon=xbar1-xbar2-t*sqrt((1/n1+1/n2)*sp);
upbon=xbar1-xbar2+t*sqrt((1/n1+1/n2)*sp);
run;
proc print;
run;

[Janus]

Use this to clean up the data (the nC2 macro will come in handy for adjusting the CIs):

data test;
input mean sd n;
cards;
10 2.5 40
15 2.0 37
08 2.3 45
17 3.4 35
11 2.9 41
;
run;

proc sql; create table test as select *, monotonic() as class from test; quit;
proc sql noprint; select count(distinct(class)) into :n from test; quit;
proc sql noprint; select fact(count(distinct(class)))/((fact(2))*(fact(count(distinct(class)) - 2))) into :nC2 from test; quit;

proc plan;
	factors test_number=&nC2. ordered
	class= 2 of &n. comb ;
	ods output plan=combinations;
	run; quit;

proc sql;
create table temp1 as
select
	c.test_number,
	c.class1,
	t.mean as mean1,
	t.sd as sd1,
	t.n as n1
from
	combinations c
	left join
	test t
	on c.class1 = t.class
; quit; 

proc sql;
create table temp2 as
select
	c.test_number,
	c.class2,
	t.mean as mean2,
	t.sd as sd2,
	t.n as n2
from
	combinations c
	left join
	test t
	on c.class2 = t.class
; quit; 

proc sql;
create table joined as
select
	t1.*,
	t2.*
from
	temp1 t1,
	temp2 t2
where
	t1.test_number = t2.test_number
order by
	t1.test_number
; quit;

[lancearmstrong1313]

I was also using PROC SQL to do the macro variables for total number of groups but had not thought of using it to create the number of distinct combinations.

I guess once the data is organized like you had it after you combine the 2 temp tables, then it will be pretty easy to actually calculate CIs. So using a non-pooled approach:

data ci_adjust;
set joined;
Lbonf_ci = (mean1 - mean2) - ((tinv(1-0.05/&nC2,n1+n2-2))*SQRT(((sd1**2)/n1)+((sd1**2)/n1)));
Ubonf_ci = (mean1 - mean2) + ((tinv(1-0.05/&nC2,n1+n2-2))*SQRT(((sd1**2)/n1)+((sd1**2)/n1)));
run; quit;