Need some help calculating standard error of multiple regression coefficients

#1
Hello. I am an undergrad student not very familiar with advanced statistics. Thus, I figured someone on this forum could help me in this regard:

The following is a webpage that calculates estimated regression coefficients for multiple linear regressions

http://people.hofstra.edu/stefan_Waner/realworld/multlinreg.html.

I would like to add on to the source code, so that I can figure out the standard error for each of the coefficients estimates in the regression. I don't understand the terminology in the source code, so I figured someone here might in order to show me how to calculate the std errors.

I would like to be able to figure this out as soon as possible. Thank you for your help.
 

Dragan

Super Moderator
#2
Hello. I am an undergrad student not very familiar with advanced statistics. Thus, I figured someone on this forum could help me in this regard:

The following is a webpage that calculates estimated regression coefficients for multiple linear regressions

http://people.hofstra.edu/stefan_Waner/realworld/multlinreg.html.

I would like to add on to the source code, so that I can figure out the standard error for each of the coefficients estimates in the regression. I don't understand the terminology in the source code, so I figured someone here might in order to show me how to calculate the std errors.

I would like to be able to figure this out as soon as possible. Thank you for your help.
You might the find this useful. Someone else asked me the (exact) same question a few weeks ago.

The standard error for a regression coefficients is:


Se(bi) = Sqrt [MSE / (SSXi * TOLi) ]

where MSE is the mean squares for error from the overall ANOVA summary, SSXi is the sum of squares for the i-th independent variable, and TOLi is the tolerance associated with the i-th independent variable.

TOLi = 1 - Ri^2, where Ri^2 is determined by regressing Xi on all the other independent variables in the model.

-- Dragan
 
#3
Thanks, I was wondering if this can be shown algebraically or numerically with some data, without matrix algebra. Since I am not sure how the numerical output is derived with javascript.

here is some sample data.
http://www.egwald.ca/statistics/electiontable2004.php
I am not sure how it goes from the data to the estimates and then to the standard deviations. If you could show me, I would really appreciate it.

Sorry, I am not very literate in advanced stat methods.
Thanks again
 
#4
Formula for calculating the standard error of the intercept for multiple regression

Thanks for the info on calculating the standard error of the multiple regression coefficients. I was wondering what formula is used for calculating the standard error of the constant term (or intercept). Any help would be greatly appreciated.

Thanks.
 

Dragan

Super Moderator
#5
Thanks for the info on calculating the standard error of the multiple regression coefficients. I was wondering what formula is used for calculating the standard error of the constant term (or intercept). Any help would be greatly appreciated.

Thanks.

In general, obtain the estimated variance-covariance matrix as (in matrix form):

S^2{b} = MSE * (X^T * X)^-1

The standard error for the intercept term, s(b0), will be the square root of the element in the first column and first row.

For a simple regression the standard error for the intercept term can be easily obtained from:

s{bo} = StdErrorReg * Sqrt [ SumX^2 / (N * SSx) ]

where StdErrorReg is the standard error of the estimate for the regression line and SSx is the sums of squares for X (the independent variable).
 
#7
Std error of intercept for multi-regression

HI

What will be the approprate equation in a multiple regression for s{bo}?


For a simple regression the standard error for the intercept term can be easily obtained from:

s{bo} = StdErrorReg * Sqrt [ SumX^2 / (N * SSx) ]

where StdErrorReg is the standard error of the estimate for the regression line and SSx is the sums of squares for X (the independent variable).[/QUOTE]
 
#8
Need help

Hi,

Could you pls show me the formula to archive the S{b0} in case of multi-regression.
I need it in an emergency.

Thanks in advance.
 

Dragan

Super Moderator
#9
Hi,

Could you pls show me the formula to archive the S{b0} in case of multi-regression.
I need it in an emergency.

Thanks in advance.
Well, it is as I said above.

I'll repeat:

In general, obtain the estimated variance-covariance matrix as (in matrix form):

S^2{b} = MSE * (X^T * X)^-1

The standard error for the intercept term, s{b0}, will be the square root of the element in the first column and first row.

I think this is clear.
 
#10
Well, it is as I said above.

I'll repeat:

In general, obtain the estimated variance-covariance matrix as (in matrix form):

S^2{b} = MSE * (X^T * X)^-1

The standard error for the intercept term, s{b0}, will be the square root of the element in the first column and first row.

I think this is clear.
Hic,
sorry for any inconvenience, but i'm not a statistican, so please show me the visual formula.
Thanks alot.
Please help, I just have 1 more day.
 

Dragan

Super Moderator
#11
Hic,
sorry for any inconvenience, but i'm not a statistican, so please show me the visual formula.
Thanks alot.
Please help, I just have 1 more day.


Here is some source code to follow. It's for a simple regression but the idea can be easily extended to multiple regression.

Code:
(* Let y be the y = {3, 4, 5, 7, 9, 9, 12} and x = {1, 3, 4, 6, 7, 8, 9}.
            Regress y on x 
      and obtain the mean square for error (MSE) which is .668965517 .. *)

(* To get the standard error use an augmented matrix for X *)

xt = {{1, 1, 1, 1, 1, 1, 1}, {1., 3, 4, 6, 7, 8, 9}};
a = MatrixForm[xt]
x = Transpose[xt];
MatrixForm[x]
Print[ "Multiply X tranpose by X"]
b = xt.x;
MatrixForm[b]
Print[ "Take the Inverse of (X^T * X)"]
c = Inverse[b];
MatrixForm[c]
mse = 0.6689655172413833;
Print[ "Now multiply each element by the MSE and take then take 
        the Square Root of each element on the diag which is all we care about here...
        I am just going to ignore the off-diag elements"]
Print[ "The standard errors are on the diag below: Intercept .7015 and for X .1160"]
u = Sqrt[mse*c];
MatrixForm[u]
 
Last edited:
#12
Here is some source code to follow. It's for a simple regression but the idea can be easily extended to multiple regression.
...
Thanks so much,
So, if i have the equation y = bo + b1*X1 + b2*X2
then, X =
(1 X11 X21)
(1 X12 X22)
(1 X13 X23)
(... )

and X^T =
(1 1 1 ...)
(X11 X12 X13 ...)
(X21 X22 X23 ...)
 
#14
Sorry for continuing the topic. Would you please specify what Mean Squared Error MSE is meant here?
There is so much notational confusion... Do you mean:
Sum of all squared residuals (residual being Observed Y minus Regression-estimated Y) divided by (n-p)?
 

Dragan

Super Moderator
#15
Sorry for continuing the topic. Would you please specify what Mean Squared Error MSE is meant here?
There is so much notational confusion... Do you mean:
Sum of all squared residuals (residual being Observed Y minus Regression-estimated Y) divided by (n-p)?
I did specify what the MSE is in my first post. And, yes, it is as you say:

MSE = SSres / df

where df = N - p

where p includes the intercept term.