Standard error of multiple regression coefficients

[davidbinyam]

I was revising my undergraduate courses of Statistics which I took before 20 years. I was exercising how to construct ANOVA table for a regression with two and three predictiors. But I couldn't obtain the standard errors of the coefficients for a test of significance becuase there is no formula in both the books of regression I bought recently. I don't want to buy a third book which again may not contain any such formula. Can anybody who knows these formulas for the standard errors leave me a message here? Thank you!!

[Dragan]

I was revising my undergraduate courses of Statistics which I took before 20 years. I was exercising how to construct ANOVA table for a regression with two and three predictiors. But I couldn't obtain the standard errors of the coefficients for a test of significance becuase there is no formula in both the books of regression I bought recently. I don't want to buy a third book which again may not contain any such formula. Can anybody who knows these formulas for the standard errors leave me a message here? Thank you!!


The standard error for a regression coefficients is:


Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]

where MSE is the mean squares for error from the overall ANOVA summaryl, SSXj is the sum of squares for the j-th independent variable, and TOLj is the tolerance associated with the j-th independent variable.

TOLj = 1 - Rj^2, where Rj^2 is determined by regressing Xj on all the other independent variables in the model.

[davidbinyam]

The standard error for a regression coefficients is:


Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]

where MSE is the mean squares for error from the overall ANOVA summaryl, SSXj is the sum of squares for the j-th independent variable, and TOLj is the tolerance associated with the j-th independent variable.

TOLj = 1 - Rj^2, where Rj^2 is determined by regressing Xj on all the other independent variables in the model.

Thank you! It works! I was also able to determine the coefficients and their standard error using matrix algebra. That happened to be more efficient. It is advantageous to know two ways of doing that.

[alex84]

Can anybody tell me what formula should be used for model with no intercept (i.e. zero constant)? (I know this model requires more attention :yup: and sanity checks).
I've tried following formula Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]
but got results different from Excel and SPPS ones.

I'm searching for a 6 hours - and I see only the formula like above. But it seems in case of model with no intercept there should be other formula.

Thanks in advance for attention :wave:!

[Dragan]

Can anybody tell me what formula should be used for model with no intercept (i.e. zero constant)? (I know this model requires more attention :yup: and sanity checks).
I've tried following formula Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]
but got results different from Excel and SPPS ones.

I'm searching for a 6 hours - and I see only the formula like above. But it seems in case of model with no intercept there should be other formula.

Thanks in advance for attention :wave:!


In terms of the calculation of the MSE you're going to have to exclude the intercept term from the degrees of freedom i.e. in simple regression it would be N - 1 instead of N -2.

Also, the SSX term should be expressed as raw sums of squares formula i.e. Sum(X^2) rather than in deviation form (because the intercept is zero).

[alex84]

In terms of the calculation of the MSE you're going to

have to exclude the intercept term from the degrees of freedom i.e. in

simple regression it would be N - 1 instead of N -2.

Also, the SSX term should be expressed as raw sums of squares formula i.e.

Sum(X^2) rather than in deviation form (because the intercept is zero).



Thanks a lot, but I still have other value :( It seems the problem is that

I'm calculating TOLj and Rj2 incorrectly.

E.g. Y={1,2,3}, X={2,3,5}.

I suppose TOL1=0 since I have no more indep. variables? Is this correct?

[Dragan]

Thanks a lot, but I still have other value :( It seems the problem is that

I'm calculating TOLj and Rj2 incorrectly.

E.g. Y={1,2,3}, X={2,3,5}.

I suppose TOL1=0 since I have no more indep. variables? Is this correct?


Yes. The tolerance, TOL, would be ONE.

[alex84]

Yes. The tolerance, TOL, would be zero.

Thanks again for prompt reply!

But in both cases - TOL=0 and TOL=1 I'm getting incorrect results.

For Y={1,2,3}, X={2,3,5} easily I've got
Sum(Xi^2,i=1..3) = 38
Sum(XiYi,i=1..3) = 23.
Beta=23/38=0.605263158 (exactly as SPSS 17 and Excel's ATP)
(skipped)
SSres=0.078947368
MSE=0.039473684 (exactly as SPSS 17 and Excel's ATP)

For TOL=1: SQRT(MSE / SUM_X^2/(1-0)) = 0.091970901
For TOL=0: can't be calculated

But both SPSS 17 and Excel ATP give me
Beta[1] = 0.605263158 Beta[1] Standard Error 0.032230128
:(

[Dragan]

Thanks again for prompt reply!

But in both cases - TOL=0 and TOL=1 I'm getting incorrect results.

For Y={1,2,3}, X={2,3,5} easily I've got
Sum(Xi^2,i=1..3) = 38
Sum(XiYi,i=1..3) = 23.
Beta=23/38=0.605263158 (exactly as SPSS 17 and Excel's ATP)
(skipped)
SSres=0.078947368
MSE=0.039473684 (exactly as SPSS 17 and Excel's ATP)

For TOL=1: SQRT(MSE / SUM_X^2/(1-0)) = 0.091970901
For TOL=0: can't be calculated

But both SPSS 17 and Excel ATP give me
Beta[1] = 0.605263158 Beta[1] Standard Error 0.032230128
:(

I’m sorry I meant to say you should ignore (or disregard) the Tolerance because you only have 1 independent variable (X) in the model.

Anyway, the correct standard error (SE) is:

SE(B) = Sqrt [ MSE / Sum(X^2) ] = Sqrt [0.039473684 / 38 ] = 0.032230128

And, that should do it :)

[alex84]

Oops. I've forgot about

the SSX term should be expressed as raw sums of squares formula i.e. Sum(X^2) rather than in deviation form
Thanks a lot!

[alex84]

Dragan,
Could you kindly suggest me formula for R2ij? I'm writing the code for my thesis and I'm wondering if there is some simpler formula, that doesn't require to make a lot of calculations. Because to get Rj^2 for each indep. variable I need to regress Xj on all the other independent variables in the model, i.e. run regression procedure N -1 times, where N is independent variables count.

Thanks in advance for reply!

[Dragan]

Dragan,
Could you kindly suggest me formula for R2ij? I'm writing the code for my thesis and I'm wondering if there is some simpler formula, that doesn't require to make a lot of calculations. Because to get Rj^2 for each indep. variable I need to regress Xj on all the other independent variables in the model, i.e. run regression procedure N -1 times, where N is independent variables count.

Thanks in advance for reply!


Is your goal just to compute the standard errors for each independent variable?...or is there something else beyond that?

[alex84]

Is your goal just to compute the standard errors for each independent variable?...or is there something else beyond that?

Just standard errors for each independent variable (and in result getting CI, that's just Tvalue(95%) * SEith) :yup:

[Dragan]

Just standard errors for each independent variable (and in result getting CI, that's just Tvalue(95%) * SEith) :yup:


Okay, what you should do is take a more computationally efficient (Matrix Algebra) approach to do this.

Read through this link below. I provided an example on how to do this.

http://talkstats.com/showthread.php?t=5056

[alex84]

Okay, what you should do is take a more computationally efficient (Matrix Algebra) approach to do this.

Read through this link below. I provided an example on how to do this.

http://talkstats.com/showthread.php?t=5056

That works! Thank you! :yup::yup::yup: