Removing outliers using SAS

#1
I know "proc means" will give me the mean and standard deviation of a set of data.

Is there any code that can remove outliers from a set of data? Outliers are defined to be values that are more or less than two standard deviations away from the mean.
 

jrai

New Member
#2
Use proc standard. This will give you the standard normal distribution of your variables. Remove the variables outside the range of -3 to 3.

This only treats univariate outliers. For multivariate outliers look at Mahalanobis distances. This can be done using proc princomp using std option in the statement. Other method is to do regress any unwanted variable (say row number) on the variables of interest (IVs) & calculate INFLUENCE statistic. This can be done in proc reg.

proc reg data=;
model row_num= a b c/ influence;
run;

This will give you a variable called leverage denoted in output as Hat diag H. Mahalanobis distance= MD=(n-1) (hat_diag_h-1/n). MD has a chi-sq distribution with degree of freedom =# of variables. This will give you the cut off beyond which you can label the observations as outliers.
 

jrai

New Member
#4
The code mentioned above is for identifying the outliers & not for deleting them. Do you want to remove the entire observation or just that cell?
 

jrai

New Member
#7
Treatment of outliers is a subjective issue. You must investigate why these outliers are there? There is not always the case to remove them but if you've to remove them delete the entire row.

You can use where condition to subset your dataset. For multivariate outliers, calculate MD & merge that with the observations in the same dataset & then subset the obs based on the cut off for MD.
 

jrai

New Member
#9
First save the influence statistics in a table:
ods output OutputStatistics=table1;

Run proc reg:

proc reg data=table;
model row_num= a b c/ influence;
model x = a b c;
run;

Close the ods output:
ods output close;

Calculate MD:

data table1(keep=md);
set table1 nobs=n; *nobs=n will store the sample size in variable n;
md=(n-1)*((hat_diag_h-1/n); *Check the exact name of the variable hat_diag_h using proc contents before using it here. I know that the label is 'Hat Diag H' but I don't
remember the variable name;
run;

Merge the two datasets:

data table;
set table;
set table1;
run;

Note that this process will be sensitive to missing values. Say your original dataset i.e. table has 10 obs but obs 1 & 2 had missing variable values. Now these will be excluded while calculating the output statistics. Now table1 will have 8 observations & the above code will automatically match md of obs 3 with obs 1 in dataset, md of obs 4 with obs2 & so on. Therefore, before running the whole program either treat missing values or exclude the observations with missing values.

Now you've MD for each observation in your dataset. Now calculate the cutoff of MD. You can quickly do this in excel with CHIINV function. =CHIINV(probability,df). Probability here should be 0.05 & df i.e. degrees of freedom equals the number of variables (=3 in our case because we've 3 variables a,b & c). Therefore, =CHIINV(0.05,3)=7.81.

Remove values greater than 7.81.

data table (drop=md);
set table (where=(md<=7.81));
run;

I'd say before removing outliers study them carefully & see if intuitively they really make sense. For example, your variables are jump distance & age. You might find that for a particular obs md>cutoff value. The jump distance for that obs=6 feet, which is not abnormal. But when you see age it equals 5 years. Definitely, 6 feet distance is not possible for a 5 year old child & hence it makes sense to discard the obs as an outlier. The whole idea is to investigate & justify whether outliers really make a case to be discarded or were these actual behaviors & need to stay in the model to enable the model to predict such instances in future.
 
Last edited:
#10
Jrai,
I know you have done a lot for me alr but I need one more help.

Right now I have a model with 1 dependent variable and 20 independent variables.

So my code is

proc reg data=Regression;
model DEPD = A B C D E F G H I J K L M N O P Q R S T;
run;

Regression is my file name, DEPD is the dependent variable and A B C D E F G H I J K L M N O P Q R S T are the independent variables.

How can I modify the code so that the regression only regresses for data within + or - 3 standard deviations of the mean for every value? All other values are ignored?

e.g. on a smaller scale

DEPD A B C D
4567 9 2 7 8
7567 12 3 9 0
9875 3 2 67 7
5635 9 12 7 88
2568 9 2 7 8
3547 9 2 7 8

So the values I want SAS to regress are

DEPD A B C D
4567 9 2 7 8
7567 12 3 9 0
2568 9 2 7 8
3547 9 2 7 8
(elimination done by eyeballing so may not be accurate)

i.e. the entire row is disregarded if one value in the column is more or less than 3 standard deviation from the mean for DEPD and A to T.

Is this possible?
 
#12
One way is to find the SDs using proc standard, merge them with DV & then do the elimination.
proc standard mean=0 std=1 data=regression out=temp(keep=depd); *This step finds the standard normal score i.e. z-score of depd;
var depd;
run;

data regression; *This step merges standard normal score with the original data set. Note: ensure that the order of the observations in the two datasets is the same;
set regression;
set temp(rename=(depd=standard_depd));
run;

proc reg data=Regression(where=(standard_depd<3 and standard_depd>-3)); *This step excludes the observations where standard deviation is more than 3 or less than -3;
model DEPD = A B C D E F G H I J K L M N O P Q R S T;
run;
 
Last edited:
#13
Alternative way is to calculate standard deviation directly:

proc means data=regression;
var depd;
output out=temp(keep=depd _stat_ where=(_stat_="STD")); *The output dataset temp will contain the standard deviation of depd;
run;

data _null_;
set temp(keep=depd);
call symput("upper_limit",3*depd); *This creates a macro variable with the upper limit;
call symput("lower_limit",-3*depd); *This creates a macro variable with the lower limit;
run;

proc reg data=Regression(where=(depd<&upper_limit and depd>&lower_limit)); *This step excludes the observations where standard deviation is more than 3 or less than -3;
model DEPD = A B C D E F G H I J K L M N O P Q R S T;
run;
 
#14
Hi everyone,
I have this set of data and I am looking for a SAS macro to find the outliers.
Can anyone help me with this please?

I would appreciate if someone can write me the macro using this set of data. Thanks

Data
0.26
0.98
0.74
0.33
0.23
0.49
0.33
0.68
1.09
1.07
0.85
1.22
0.20
0.35
0.30
0.38
0.31
0.67
 

hlsmith

Omega Contributor
#15
As for the original post, jrai provided some great information. However, you never really say why you wanted to eliminate the outlying values - I wonder what their purpose was and whether they realize that once removing those values > or < 3 stds, that now there are new observations in the set that are < or > 3 stds away from the mean, this may be beside the point.

canadian, what is the purpose of finding the outliers and how are you defining outliers. Also, I don't think you really need a macro to do this. You can review the previously posted code to come up with a solution.
 

hlsmith

Omega Contributor
#16
Syntax - taking from previous code on this thread:

Code:
[SIZE=2][FONT=Courier New][COLOR=#000080][SIZE=2][FONT=Courier New][COLOR=#000080][B][SIZE=2][FONT=Courier New][COLOR=#000080]data[/COLOR][/FONT][/SIZE][/B][/COLOR][/FONT][/SIZE][/COLOR][/FONT][/SIZE][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] canadian;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]input[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]Datalines[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[SIZE=2][FONT=Courier New]0.26[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.98[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.74[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.33[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.23[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.49[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.33[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.68[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]1.09[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]1.07[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.85[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]1.22[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.20[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.35[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.30[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.38[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.31[/FONT][/SIZE]
[SIZE=2][FONT=Courier New]0.67[/FONT][/SIZE]
[SIZE=2][FONT=Courier New];[/FONT][/SIZE]
[/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT][/SIZE][/FONT]
 
[B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]proc [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]means [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]data[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]=canadian;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]var[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]output[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]out[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]=temp (keep=bacon _stat_ where=(_stat_=[/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080]"STD"[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]));[/SIZE][/FONT]
[/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT][/SIZE][/FONT]
 
[B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]data[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]_null_[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]set[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] temp(keep=bacon);[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]call[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] symput([/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080]"upper_limit"[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2],[/SIZE][/FONT][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080]3[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]*bacon); [/SIZE][/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]call[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] symput([/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080]"lower_limit"[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2],-[/SIZE][/FONT][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080]3[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]*bacon); [/SIZE][/FONT][/SIZE][/FONT]
[B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT][/SIZE][/FONT]
 
[B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]data[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] canadian_2;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]set[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] canadian;[/SIZE][/FONT]
[SIZE=2][FONT=Courier New]Outlier = [/FONT][/SIZE][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080]1[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]if[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon <&upper_limit and bacon>&lower_limit [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]then[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] Outlier = [/SIZE][/FONT][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080]0[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[SIZE=2][FONT=Courier New]extra_bacon = log(Bacon); [/FONT][/SIZE][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000]/*extra - not needed */[/COLOR][/SIZE][/FONT]
[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][/SIZE][/FONT]
 

hlsmith

Omega Contributor
#17
Wouldn't let me post all at once - see below for subsequent code:
Code:
[FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000]/* Extra Syntax (not needed)*/[/COLOR][/SIZE][/FONT]
[SIZE=2][FONT=Courier New][COLOR=#008000]/*Data not normally distributed */[/COLOR][/FONT][/SIZE]
[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]proc [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]univariate data=canadian_2 [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]var[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]histogram[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon / [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]probplot[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon / [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000]/* Data not normally distributed after removing outliers */[/COLOR][/SIZE][/FONT]
[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]proc [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]univariate data=canadian_2 [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]where[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] outlier = [/SIZE][/FONT][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080][FONT=Courier New][SIZE=2][COLOR=#008080]0[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]var[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]histogram[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon / [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]probplot[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] bacon / [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000][FONT=Courier New][SIZE=2][COLOR=#008000]/*Data normally distributed */[/COLOR][/SIZE][/FONT]
[SIZE=2][FONT=Courier New][COLOR=#008000]/*Histogram is not great, but much better than removing observations */[/COLOR][/FONT][/SIZE]
[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]proc [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]univariate [/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]data[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]=canadian_2 [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]var[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] extra_bacon;[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]histogram[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] extra_bacon / [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]probplot[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] extra_bacon / [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]normal[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT][B][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080][FONT=Courier New][SIZE=2][COLOR=#000080]run[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/B][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2];[/SIZE][/FONT]
[/SIZE][/FONT]
 

noetsi

Fortran must die
#18
As hlsmith noted when you remove outliers there are commonly new outlier (in part because the outliers you removed masked other outliers). But a more basic question is if you should remove any outlier. And that depends on why it is an outlier. Commonly outliers occur because something is wrong in your model, you specified the wrong form or the wrong variables, and you should seek to understand why the outlier is occuring to suggest changes in the model.