+ Reply to Thread
Results 1 to 15 of 15

Thread: Standard error of multiple regression coefficients

  1. #1
    Points: 3,583, Level: 37
    Level completed: 56%, Points required for next Level: 67

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Standard error of multiple regression coefficients




    I was revising my undergraduate courses of Statistics which I took before 20 years. I was exercising how to construct ANOVA table for a regression with two and three predictiors. But I couldn't obtain the standard errors of the coefficients for a test of significance becuase there is no formula in both the books of regression I bought recently. I don't want to buy a third book which again may not contain any such formula. Can anybody who knows these formulas for the standard errors leave me a message here? Thank you!!

  2. #2
    Super Moderator
    Points: 10,218, Level: 67
    Level completed: 43%, Points required for next Level: 232
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,838
    Thanks
    0
    Thanked 156 Times in 139 Posts
    Quote Originally Posted by davidbinyam View Post
    I was revising my undergraduate courses of Statistics which I took before 20 years. I was exercising how to construct ANOVA table for a regression with two and three predictiors. But I couldn't obtain the standard errors of the coefficients for a test of significance becuase there is no formula in both the books of regression I bought recently. I don't want to buy a third book which again may not contain any such formula. Can anybody who knows these formulas for the standard errors leave me a message here? Thank you!!

    The standard error for a regression coefficients is:


    Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]

    where MSE is the mean squares for error from the overall ANOVA summaryl, SSXj is the sum of squares for the j-th independent variable, and TOLj is the tolerance associated with the j-th independent variable.

    TOLj = 1 - Rj^2, where Rj^2 is determined by regressing Xj on all the other independent variables in the model.

  3. #3
    Points: 3,583, Level: 37
    Level completed: 56%, Points required for next Level: 67

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dragan View Post
    The standard error for a regression coefficients is:


    Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]

    where MSE is the mean squares for error from the overall ANOVA summaryl, SSXj is the sum of squares for the j-th independent variable, and TOLj is the tolerance associated with the j-th independent variable.

    TOLj = 1 - Rj^2, where Rj^2 is determined by regressing Xj on all the other independent variables in the model.
    Thank you! It works! I was also able to determine the coefficients and their standard error using matrix algebra. That happened to be more efficient. It is advantageous to know two ways of doing that.

  4. #4
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Can anybody tell me what formula should be used for model with no intercept (i.e. zero constant)? (I know this model requires more attention and sanity checks).
    I've tried following formula Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]
    but got results different from Excel and SPPS ones.

    I'm searching for a 6 hours - and I see only the formula like above. But it seems in case of model with no intercept there should be other formula.

    Thanks in advance for attention !

  5. #5
    Super Moderator
    Points: 10,218, Level: 67
    Level completed: 43%, Points required for next Level: 232
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,838
    Thanks
    0
    Thanked 156 Times in 139 Posts
    Quote Originally Posted by alex84 View Post
    Can anybody tell me what formula should be used for model with no intercept (i.e. zero constant)? (I know this model requires more attention and sanity checks).
    I've tried following formula Se(bj) = Sqrt [MSE / (SSXj * TOLj) ]
    but got results different from Excel and SPPS ones.

    I'm searching for a 6 hours - and I see only the formula like above. But it seems in case of model with no intercept there should be other formula.

    Thanks in advance for attention !

    In terms of the calculation of the MSE you're going to have to exclude the intercept term from the degrees of freedom i.e. in simple regression it would be N - 1 instead of N -2.

    Also, the SSX term should be expressed as raw sums of squares formula i.e. Sum(X^2) rather than in deviation form (because the intercept is zero).

  6. #6
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dragan View Post
    In terms of the calculation of the MSE you're going to

    have to exclude the intercept term from the degrees of freedom i.e. in

    simple regression it would be N - 1 instead of N -2.

    Also, the SSX term should be expressed as raw sums of squares formula i.e.

    Sum(X^2) rather than in deviation form (because the intercept is zero).
    Thanks a lot, but I still have other value It seems the problem is that

    I'm calculating TOLj and Rj2 incorrectly.

    E.g. Y={1,2,3}, X={2,3,5}.

    I suppose TOL1=0 since I have no more indep. variables? Is this correct?

  7. #7
    Super Moderator
    Points: 10,218, Level: 67
    Level completed: 43%, Points required for next Level: 232
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,838
    Thanks
    0
    Thanked 156 Times in 139 Posts
    Quote Originally Posted by alex84 View Post
    Thanks a lot, but I still have other value It seems the problem is that

    I'm calculating TOLj and Rj2 incorrectly.

    E.g. Y={1,2,3}, X={2,3,5}.

    I suppose TOL1=0 since I have no more indep. variables? Is this correct?

    Yes. The tolerance, TOL, would be ONE.
    Last edited by Dragan; 12-06-2009 at 03:02 PM. Reason: correct error

  8. #8
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dragan View Post
    Yes. The tolerance, TOL, would be zero.
    Thanks again for prompt reply!

    But in both cases - TOL=0 and TOL=1 I'm getting incorrect results.

    For Y={1,2,3}, X={2,3,5} easily I've got
    Sum(Xi^2,i=1..3) = 38
    Sum(XiYi,i=1..3) = 23.
    Beta=23/38=0.605263158 (exactly as SPSS 17 and Excel's ATP)
    (skipped)
    SSres=0.078947368
    MSE=0.039473684 (exactly as SPSS 17 and Excel's ATP)

    For TOL=1: SQRT(MSE / SUM_X^2/(1-0)) = 0.091970901
    For TOL=0: can't be calculated

    But both SPSS 17 and Excel ATP give me
    Beta[1] = 0.605263158 Beta[1] Standard Error 0.032230128
    Last edited by alex84; 12-06-2009 at 11:46 AM.

  9. #9
    Super Moderator
    Points: 10,218, Level: 67
    Level completed: 43%, Points required for next Level: 232
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,838
    Thanks
    0
    Thanked 156 Times in 139 Posts
    Quote Originally Posted by alex84 View Post
    Thanks again for prompt reply!

    But in both cases - TOL=0 and TOL=1 I'm getting incorrect results.

    For Y={1,2,3}, X={2,3,5} easily I've got
    Sum(Xi^2,i=1..3) = 38
    Sum(XiYi,i=1..3) = 23.
    Beta=23/38=0.605263158 (exactly as SPSS 17 and Excel's ATP)
    (skipped)
    SSres=0.078947368
    MSE=0.039473684 (exactly as SPSS 17 and Excel's ATP)

    For TOL=1: SQRT(MSE / SUM_X^2/(1-0)) = 0.091970901
    For TOL=0: can't be calculated

    But both SPSS 17 and Excel ATP give me
    Beta[1] = 0.605263158 Beta[1] Standard Error 0.032230128
    Im sorry I meant to say you should ignore (or disregard) the Tolerance because you only have 1 independent variable (X) in the model.

    Anyway, the correct standard error (SE) is:

    SE(B) = Sqrt [ MSE / Sum(X^2) ] = Sqrt [0.039473684 / 38 ] = 0.032230128

    And, that should do it

  10. #10
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Oops. I've forgot about
    the SSX term should be expressed as raw sums of squares formula i.e. Sum(X^2) rather than in deviation form
    Thanks a lot!

  11. #11
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Dragan,
    Could you kindly suggest me formula for R2ij? I'm writing the code for my thesis and I'm wondering if there is some simpler formula, that doesn't require to make a lot of calculations. Because to get Rj^2 for each indep. variable I need to regress Xj on all the other independent variables in the model, i.e. run regression procedure N -1 times, where N is independent variables count.

    Thanks in advance for reply!

  12. #12
    Super Moderator
    Points: 10,218, Level: 67
    Level completed: 43%, Points required for next Level: 232
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,838
    Thanks
    0
    Thanked 156 Times in 139 Posts
    Quote Originally Posted by alex84 View Post
    Dragan,
    Could you kindly suggest me formula for R2ij? I'm writing the code for my thesis and I'm wondering if there is some simpler formula, that doesn't require to make a lot of calculations. Because to get Rj^2 for each indep. variable I need to regress Xj on all the other independent variables in the model, i.e. run regression procedure N -1 times, where N is independent variables count.

    Thanks in advance for reply!

    Is your goal just to compute the standard errors for each independent variable?...or is there something else beyond that?

  13. #13
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Dragan View Post
    Is your goal just to compute the standard errors for each independent variable?...or is there something else beyond that?
    Just standard errors for each independent variable (and in result getting CI, that's just Tvalue(95%) * SEith)

  14. #14
    Super Moderator
    Points: 10,218, Level: 67
    Level completed: 43%, Points required for next Level: 232
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    1,838
    Thanks
    0
    Thanked 156 Times in 139 Posts
    Quote Originally Posted by alex84 View Post
    Just standard errors for each independent variable (and in result getting CI, that's just Tvalue(95%) * SEith)

    Okay, what you should do is take a more computationally efficient (Matrix Algebra) approach to do this.

    Read through this link below. I provided an example on how to do this.

    http://talkstats.com/showthread.php?t=5056

  15. #15
    Points: 2,106, Level: 27
    Level completed: 71%, Points required for next Level: 44

    Posts
    14
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Quote Originally Posted by Dragan View Post
    Okay, what you should do is take a more computationally efficient (Matrix Algebra) approach to do this.

    Read through this link below. I provided an example on how to do this.

    http://talkstats.com/showthread.php?t=5056
    That works! Thank you!

+ Reply to Thread

           




Similar Threads

  1. Std Error of nonlinear regression coefficients
    By Dolphin72 in forum Regression Analysis
    Replies: 0
    Last Post: 03-10-2011, 08:11 PM
  2. Replies: 15
    Last Post: 09-09-2010, 06:05 PM
  3. Standard Error of Coefficients in Multiple Regression
    By JoeA838 in forum Regression Analysis
    Replies: 1
    Last Post: 10-26-2009, 03:49 PM
  4. Replies: 0
    Last Post: 10-23-2008, 11:49 AM
  5. standard error of ratio of coefficients
    By dragonflyFZX in forum Statistics
    Replies: 2
    Last Post: 02-05-2007, 12:48 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats