+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 16 to 25 of 25

Thread: how to prove COV(y bar, estimated beta1) in linear regression equals 0

  1. #16
    Points: 2,350, Level: 29
    Level completed: 34%, Points required for next Level: 100

    Location
    Philadelphia
    Posts
    46
    Thanks
    1
    Thanked 2 Times in 1 Post

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals




    Quote Originally Posted by Dragan View Post
    You can find the proof of this in other threads here - I did it not too long ago. This question has come up a few times. It goes like this (I'll be a little neater this time):


    Cov\left [ b_{0} ,b_{1}\right ]=E\left [ \left ( b_{0}-E\left [  b_{0}\right ] \right )\left ( b_{1}-E\left [b _{1} \right ] \right ) \right ]

    =E\left [ \left ( b_{0}-\beta _{0} \right )\left (b _{1} -\beta _{1}\right ) \right ]

    =-\bar{X}E\left [ b_{1}-\beta _{1} \right ]^{2}

    =-\bar{X}Var\left [b _{1} \right ]



    =-\bar{X}\frac{\sigma ^{2}}{SS_{X}}

    where I am making use of

    b_{0}=\bar{Y}-b_{1}\bar{X}

    E\left [b _{0} \right ]=\bar{Y}-\beta _{1}\bar{X}

    giving

    \left (b _{0}-E\left [b _{0} \right ] \right )=-\bar{X}\left ( b_{1}-\beta _{1} \right ).

    The last part looks suspicious. Shouldn't it be E(Ybar) instead of Ybar??
    Confused.

  2. #17
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    Quote Originally Posted by hehe1223 View Post
    The last part looks suspicious. Shouldn't it be E(Ybar) instead of Ybar??
    Confused.
    What I wrote is correct hehe1223.

    More specifically, the sample regression function can be expressed as:

    Y_{i}=\hat{\beta _{0}}+\hat{\beta _{1}}X_{i}+e_{i}

    which estimates the population regression function written as:

    Y_{i}=\beta _{0}+\beta _{1}X_{i}+u_{i}.

    As such, it is legitimate to write,

    \bar{Y}=\beta _{0}+\beta _{1}\bar{X}+\bar{u}.

    Now, we know that the estimate of the intercept term is an unbiased estimate i.e.

    E\left [\hat{\beta _{0}}\right ]=\beta _{0}

    Therefore,

    \beta _{0}=\bar{Y}-\beta _{1}\bar{X}

    where the mean of the stochastic disturbance term is (by assumption) zero.

  3. #18
    Points: 2,350, Level: 29
    Level completed: 34%, Points required for next Level: 100

    Location
    Philadelphia
    Posts
    46
    Thanks
    1
    Thanked 2 Times in 1 Post

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    Quote Originally Posted by Dragan View Post
    What I wrote is correct hehe1223.

    More specifically, the sample regression function can be expressed as:

    Y_{i}=\hat{\beta _{0}}+\hat{\beta _{1}}X_{i}+e_{i}

    which estimates the population regression function written as:

    Y_{i}=\beta _{0}+\beta _{1}X_{i}+u_{i}.

    As such, it is legitimate to write,

    \bar{Y}=\beta _{0}+\beta _{1}\bar{X}+\bar{u}.

    Now, we know that the estimate of the intercept term is an unbiased estimate i.e.

    E\left [\hat{\beta _{0}}\right ]=\beta _{0}

    Therefore,

    \beta _{0}=\bar{Y}-\beta _{1}\bar{X}

    where the mean of the stochastic disturbance term is (by assumption) zero.
    I don't think you are correct. When there are finite number of observations. Errors do not sum up to 0!

  4. #19
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    Quote Originally Posted by hehe1223 View Post
    I don't think you are correct. When there are finite number of observations. Errors do not sum up to 0!
    Okay, let me back-up and repeat.

    That is, again, it is as I said:


    Cov\left [ b_{0} ,b_{1}\right ]=E\left [ \left ( b_{0}-E\left [  b_{0}\right ] \right )\left ( b_{1}-E\left [b _{1} \right ] \right ) \right ]

    =E\left [ \left ( b_{0}-\beta _{0} \right )\left (b _{1} -\beta _{1}\right ) \right ]

    =-\bar{X}E\left [ b_{1}-\beta _{1} \right ]^{2}

    =-\bar{X}Var\left [b _{1} \right ]

    where I am making use of

    b_{0}=\bar{Y}-b_{1}\bar{X}

    E\left [b _{0} \right ]=\bar{Y}-\beta _{1}\bar{X}

    giving

    \left (b _{0}-E\left [b _{0} \right ] \right )=-\bar{X}\left ( b_{1}-\beta _{1} \right ).



    Now, if you want to provide a different algebraic proof, then please do so and I'll look at, Mkay.

  5. #20
    Points: 2,350, Level: 29
    Level completed: 34%, Points required for next Level: 100

    Location
    Philadelphia
    Posts
    46
    Thanks
    1
    Thanked 2 Times in 1 Post

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    The problem is that you cannot assume E(Ybar)=Ybar, error terms do not sum to 0.
    I later found a way to prove Cov(ybar, slopehat) = 0
    if you express slopehat as Sxy/Sxx, you get numerator in terms of sum of (xi-xbar)*yi, express ybar as sum of yi/n. Cov(sum of yi, yi) = var(yi)=variance of error, because all yi are independent. make use of sum of (xi-xbar)=0, you will get final answer as 0.

  6. The Following 2 Users Say Thank You to hehe1223 For This Useful Post:

    fouriertransformer (05-25-2012), lemonlime (04-05-2013)

  7. #21
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    Quote Originally Posted by hehe1223 View Post
    The problem is that you cannot assume E(Ybar)=Ybar, error terms do not sum to 0.
    I later found a way to prove Cov(ybar, slopehat) = 0
    if you express slopehat as Sxy/Sxx, you get numerator in terms of sum of (xi-xbar)*yi, express ybar as sum of yi/n. Cov(sum of yi, yi) = var(yi)=variance of error, because all yi are independent. make use of sum of (xi-xbar)=0, you will get final answer as 0.
    I'll repeat, please provide me with your proof of the following (which is true) and I will look at it, Mkay.

    Here it is:

    Cov\left [ b_{0} ,b_{1}\right ]=E\left [ \left ( b_{0}-E\left [  b_{0}\right ] \right )\left ( b_{1}-E\left [b _{1} \right ] \right ) \right ]

    =-\bar{X}Var\left [b _{1} \right ].

    Now, show me your algebraic proof.

  8. #22
    Points: 3,680, Level: 38
    Level completed: 20%, Points required for next Level: 120

    Posts
    1
    Thanks
    1
    Thanked 3 Times in 1 Post

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    This thread is rather old, but I came across it while trying to answer the same question and I thought I should provide the details for others who may inquire in the future. User hehe1223 gives the correct approach; namely:

    First express the estimate of the slope \hat{\beta}_1 as
    \hat{\beta}_1 = \frac{\sum_{i = 1}^n x_iy_i - n \bar{x} \bar{y}}{\sum_{i = 1}^n x_i^2 - n \bar{x}^2}
= C  \left( \sum_{i = 1}^n x_iy_i - \bar{x} \sum_{i = 1}^n y_i \right) = C \sum_{i = 1}^n (x_i - \bar{x})y_i,
    where C = \left( \sum_{i = 1}^n x_i^2 - n \bar{x}^2 \right)^{-1}. Now, as hehe1223 suggests, express \bar{y} as a sum of y_i's and use the bilinearity of the covariance operator:
    \text{Cov}(\bar{y}, \hat{\beta}_1) = \text{Cov} \left( \frac{1}{n} \sum_{i = 1}^n y_i, C \sum_{j = 1}^n (x_j - \bar{x})y_j \right) = \frac{C}{n} \sum_{i = 1}^n \sum_{j = 1}^n (x_i - \bar{x}) \text{Cov}(y_i, y_j).
    (Note that we are implicitly conditioning on the x_i's, hence we may treat them as constants, i.e. non-random.) Now, the y_i's are usually assumed to be independent; hence \text{Cov}(y_i, y_j)  = 0 for i \neq j so that only the terms in the double sum with i = j survive, and the expression becomes a single sum. The usual assumption is that the error terms are homoscedastic, which is a fancy way of saying that they all have the same variance, say \sigma^2. Then \text{Cov}(y_i,y_i) = \text{Var}(y_i) = \sigma^2, so that we have
    \text{Cov}(\bar{y}, \hat{\beta}_1)  = \frac{C}{n} \sum_{i = 1}^n \sum_{j = 1}^n (x_i - \bar{x}) \text{Cov}(y_i, y_j) = \frac{C}{n} \sum_{i = 1}^n (x_i - \bar{x}) \text{Cov}(y_i, y_i) = \frac{C}{n} \sigma^2 \sum_{i = 1}^n (x_i - \bar{x}) = 0,
    where we have used the fact that \sum_{i = 1}^n (x_i - \bar{x}) = 0 for the last equality. I hope someone finds this helpful.
    Note that the equality that Dragan asserts now follows from this. In their notation (see previous posts):
    \text{Cov}(b_0, b_1) = \text{Cov}(\bar{Y} - b_1 \bar{X}, b_1) = \text{Cov}(\bar{Y}, b_1) - \bar{X} \text{Cov}(b_1, b_1) =  - \bar{X} \text{Var}(b_1).

  9. The Following 3 Users Say Thank You to fouriertransformer For This Useful Post:

    lemonlime (04-05-2013), M!ss Moon (04-26-2013), runrunrun (09-10-2015)

  10. #23
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: how to prove COV(y bar, estimated beta1) in linear regression equals 0

    "This thread is rather old, but I came across it..I hope someone finds this helpful....Note that the equality that Dragan asserts now follows from this."

    What I wrote in 2008 is correct and still holds today. In short, stop making yourself look like an idiot and wasting my time, Mkay.

  11. #24
    Points: 6, Level: 1
    Level completed: 11%, Points required for next Level: 44

    Posts
    1
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Re: how to prove COV(estimated beta0, estimated beta1) in linear regression is equals

    Thanks for the details fouriertransformer! I had the hint, like user hehe1223 said, to show cov(ybar, beta1_hat) = 0 first, but it was not clear how to show it. You make it very clear and easy to follow. Thanks!

  12. #25
    Points: 4, Level: 1
    Level completed: 7%, Points required for next Level: 46

    Posts
    1
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: how to prove COV(y bar, estimated beta1) in linear regression equals 0


    Quote Originally Posted by Dragan View Post
    "This thread is rather old, but I came across it..I hope someone finds this helpful....Note that the equality that Dragan asserts now follows from this."

    What I wrote in 2008 is correct and still holds today. In short, stop making yourself look like an idiot and wasting my time, Mkay.
    I am surprised a super moderator will reply in this way plus the fact the you are the one who made a mistake in the proof. Just look at the key part of your proof: beta_0 = y^bar-beta_1*x^bar, Y^bar is the only random variable in this equation, how can you equate a unknown constant with a random variable? Yes, part of what you wrote in 2008 is correct, and that is the conclusion part. And hehe1223 pointed your mistake out correctly for you. No one is wasting your time!

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2

           




Similar Threads

  1. Regression Estimated variance and Chi-square distribution
    By saaagar in forum Regression Analysis
    Replies: 0
    Last Post: 11-21-2010, 04:48 AM
  2. R^2 for linear multiple linear regression
    By raver_68 in forum Regression Analysis
    Replies: 5
    Last Post: 05-12-2010, 04:25 PM
  3. Replies: 0
    Last Post: 02-23-2009, 03:55 PM
  4. Testing if median of a sample equals zero
    By mimes in forum Statistics
    Replies: 2
    Last Post: 05-13-2008, 02:29 PM
  5. My F Value equals zero
    By sturgeon in forum Psychology Statistics
    Replies: 1
    Last Post: 09-07-2006, 12:43 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats