Standard error of multiple regression coefficients

#1
I was revising my undergraduate courses of Statistics which I took before 20 years. I was exercising how to construct ANOVA table for a regression with two and three predictiors. But I couldn't obtain the standard errors of the coefficients for a test of significance becuase there is no formula in both the books of regression I bought recently. I don't want to buy a third book which again may not contain any such formula. Can anybody who knows these formulas for the standard errors leave me a message here? Thank you!!
 

Dragan

Super Moderator
#2
I was revising my undergraduate courses of Statistics which I took before 20 years. I was exercising how to construct ANOVA table for a regression with two and three predictiors. But I couldn't obtain the standard errors of the coefficients for a test of significance becuase there is no formula in both the books of regression I bought recently. I don't want to buy a third book which again may not contain any such formula. Can anybody who knows these formulas for the standard errors leave me a message here? Thank you!!

The standard error for a regression coefficients is:


Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]

where MSE is the mean squares for error from the overall ANOVA summaryl, SSXj is the sum of squares for the j-th independent variable, and TOLj is the tolerance associated with the j-th independent variable.

TOLj = 1 - Rj^2, where Rj^2 is determined by regressing Xj on all the other independent variables in the model.
 
#3
The standard error for a regression coefficients is:


Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]

where MSE is the mean squares for error from the overall ANOVA summaryl, SSXj is the sum of squares for the j-th independent variable, and TOLj is the tolerance associated with the j-th independent variable.

TOLj = 1 - Rj^2, where Rj^2 is determined by regressing Xj on all the other independent variables in the model.
Thank you! It works! I was also able to determine the coefficients and their standard error using matrix algebra. That happened to be more efficient. It is advantageous to know two ways of doing that.
 
#4
Can anybody tell me what formula should be used for model with no intercept (i.e. zero constant)? (I know this model requires more attention :yup: and sanity checks).
I've tried following formula Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]
but got results different from Excel and SPPS ones.

I'm searching for a 6 hours - and I see only the formula like above. But it seems in case of model with no intercept there should be other formula.

Thanks in advance for attention :wave:!
 

Dragan

Super Moderator
#5
Can anybody tell me what formula should be used for model with no intercept (i.e. zero constant)? (I know this model requires more attention :yup: and sanity checks).
I've tried following formula Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]
but got results different from Excel and SPPS ones.

I'm searching for a 6 hours - and I see only the formula like above. But it seems in case of model with no intercept there should be other formula.

Thanks in advance for attention :wave:!

In terms of the calculation of the MSE you're going to have to exclude the intercept term from the degrees of freedom i.e. in simple regression it would be N - 1 instead of N -2.

Also, the SSX term should be expressed as raw sums of squares formula i.e. Sum(X^2) rather than in deviation form (because the intercept is zero).
 
#6
In terms of the calculation of the MSE you're going to

have to exclude the intercept term from the degrees of freedom i.e. in

simple regression it would be N - 1 instead of N -2.

Also, the SSX term should be expressed as raw sums of squares formula i.e.

Sum(X^2) rather than in deviation form (because the intercept is zero).
Thanks a lot, but I still have other value :( It seems the problem is that

I'm calculating TOLj and Rj2 incorrectly.

E.g. Y={1,2,3}, X={2,3,5}.

I suppose TOL1=0 since I have no more indep. variables? Is this correct?
 

Dragan

Super Moderator
#7
Thanks a lot, but I still have other value :( It seems the problem is that

I'm calculating TOLj and Rj2 incorrectly.

E.g. Y={1,2,3}, X={2,3,5}.

I suppose TOL1=0 since I have no more indep. variables? Is this correct?

Yes. The tolerance, TOL, would be ONE.
 
Last edited:
#8
Yes. The tolerance, TOL, would be zero.
Thanks again for prompt reply!

But in both cases - TOL=0 and TOL=1 I'm getting incorrect results.

For Y={1,2,3}, X={2,3,5} easily I've got
Sum(Xi^2,i=1..3) = 38
Sum(XiYi,i=1..3) = 23.
Beta=23/38=0.605263158 (exactly as SPSS 17 and Excel's ATP)
(skipped)
SSres=0.078947368
MSE=0.039473684 (exactly as SPSS 17 and Excel's ATP)

For TOL=1: SQRT(MSE / SUM_X^2/(1-0)) = 0.091970901
For TOL=0: can't be calculated

But both SPSS 17 and Excel ATP give me
Beta[1] = 0.605263158 Beta[1] Standard Error 0.032230128
:(
 
Last edited:

Dragan

Super Moderator
#9
Thanks again for prompt reply!

But in both cases - TOL=0 and TOL=1 I'm getting incorrect results.

For Y={1,2,3}, X={2,3,5} easily I've got
Sum(Xi^2,i=1..3) = 38
Sum(XiYi,i=1..3) = 23.
Beta=23/38=0.605263158 (exactly as SPSS 17 and Excel's ATP)
(skipped)
SSres=0.078947368
MSE=0.039473684 (exactly as SPSS 17 and Excel's ATP)

For TOL=1: SQRT(MSE / SUM_X^2/(1-0)) = 0.091970901
For TOL=0: can't be calculated

But both SPSS 17 and Excel ATP give me
Beta[1] = 0.605263158 Beta[1] Standard Error 0.032230128
:(
I’m sorry I meant to say you should ignore (or disregard) the Tolerance because you only have 1 independent variable (X) in the model.

Anyway, the correct standard error (SE) is:

SE(B) = Sqrt [ MSE / Sum(X^2) ] = Sqrt [0.039473684 / 38 ] = 0.032230128

And, that should do it :)
 
#11
Dragan,
Could you kindly suggest me formula for R2ij? I'm writing the code for my thesis and I'm wondering if there is some simpler formula, that doesn't require to make a lot of calculations. Because to get Rj^2 for each indep. variable I need to regress Xj on all the other independent variables in the model, i.e. run regression procedure N -1 times, where N is independent variables count.

Thanks in advance for reply!
 

Dragan

Super Moderator
#12
Dragan,
Could you kindly suggest me formula for R2ij? I'm writing the code for my thesis and I'm wondering if there is some simpler formula, that doesn't require to make a lot of calculations. Because to get Rj^2 for each indep. variable I need to regress Xj on all the other independent variables in the model, i.e. run regression procedure N -1 times, where N is independent variables count.

Thanks in advance for reply!

Is your goal just to compute the standard errors for each independent variable?...or is there something else beyond that?