+ Reply to Thread
Results 1 to 12 of 12

Thread: OLS vs solving a system of linear equaions

  1. #1
    Points: 5,083, Level: 45
    Level completed: 67%, Points required for next Level: 67

    Posts
    109
    Thanks
    19
    Thanked 3 Times in 3 Posts

    OLS vs solving a system of linear equaions




    If i want to solve y=Xb, where y is a vector of observations and X a matrix of independent variables i could use ordinary least squares to estimate the vector b, giving:

    b = (X^TX)^-1 X^T y

    But i came across some matlab code which claims to solve by OLS but actually just solves it as a system of linear equations:

    b = X^-1 y

    Are these really the same thing? Doesn't OLS assume that there is some variance in y while the other just solves the system in a deterministic manner?

    I've read on wikipedia that the matrix (X^TX)^-1 X^T, is also referred to as the Moore-Penrose pseudo-inverse so are they comparable depending on the rank of the X?

    p.s. anyone else having trouble with latex at the moment?

  2. #2
    Points: 1,974, Level: 26
    Level completed: 74%, Points required for next Level: 26

    Location
    New Zealand
    Posts
    227
    Thanks
    3
    Thanked 48 Times in 47 Posts

    Re: OLS vs solving a system of linear equaions

    The second is where you have the same number of equations as unknowns - solving linear equations. The first is where you have more equations than unknowns - linear regression. So, not really the same thing (except perhaps as a limiting case).

  3. #3
    Points: 5,083, Level: 45
    Level completed: 67%, Points required for next Level: 67

    Posts
    109
    Thanks
    19
    Thanked 3 Times in 3 Posts

    Re: OLS vs solving a system of linear equaions

    Yes, that's what i thought. But what is baffling me is that the system must be over-determined so i find it extremely unlikely that the latter should have any solutions, but the code does return a answer.

  4. #4
    Points: 1,974, Level: 26
    Level completed: 74%, Points required for next Level: 26

    Location
    New Zealand
    Posts
    227
    Thanks
    3
    Thanked 48 Times in 47 Posts

    Re: OLS vs solving a system of linear equaions

    You are solving a different set of equations - the ones you get after you get after you find the minimum squared error. You find the sum of the squares of the errors by adding all the (y-a-bx -cz ...)^2 type terms. Then after differentiating with respect to a, b, c, ... and setting them each to zero, you get a new system of equations of the right rank. This is all done by that complicated matrix expression.

  5. #5
    Points: 5,083, Level: 45
    Level completed: 67%, Points required for next Level: 67

    Posts
    109
    Thanks
    19
    Thanked 3 Times in 3 Posts

    Re: OLS vs solving a system of linear equaions

    That's what i would do.

    But the code i have doesn't do that:

    Code: 
    U = pinv(M)*Y;
    res = norm(M*X-Y,'fro');
    They're solving Y = MU

    Yet the author calls it OLS, and a solution is returned. That's what has me confused. Does Matlab automatically perform OLS on an over-determined system perhaps?

  6. #6
    Devorador de queso
    Points: 95,819, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,935
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: OLS vs solving a system of linear equaions

    Does pinv stand for pseudoinverse
    I don't have emotions and sometimes that makes me very sad.

  7. #7
    Points: 5,083, Level: 45
    Level completed: 67%, Points required for next Level: 67

    Posts
    109
    Thanks
    19
    Thanked 3 Times in 3 Posts

    Re: OLS vs solving a system of linear equaions

    Yes. Apparently matlab has two pseudo-inverses - this one is the Moore-Penrose inverse.

  8. #8
    Devorador de queso
    Points: 95,819, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,935
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: OLS vs solving a system of linear equaions

    So it's doing what one would expect.

    Also about the latex issue... yeah the board has experienced some degradation in terms of features lately. It used to be that using [math] [/math] instead of [latex] [/latex] would allow you to do most latex but that isn't doing so well anymore. The search functionality is also broken. quark has said that at some point in the future he is going to upgrade the forum software so these features should be fixed when that happens but unfortunately I haven't seen quark around for some time now.
    I don't have emotions and sometimes that makes me very sad.

  9. #9
    Points: 5,083, Level: 45
    Level completed: 67%, Points required for next Level: 67

    Posts
    109
    Thanks
    19
    Thanked 3 Times in 3 Posts

    Re: OLS vs solving a system of linear equaions

    What would you expect?

    Is there any merit to the author calling it OLS?

    If the system is over-determined how is matlab returning a solution? The matrix M has dimension 891x13.

    Does matlab automatically perform OLS on an over-determined system?

  10. #10
    Devorador de queso
    Points: 95,819, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,935
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: OLS vs solving a system of linear equaions

    Last edited by Dason; 09-20-2017 at 07:04 AM.
    I don't have emotions and sometimes that makes me very sad.

  11. The Following User Says Thank You to Dason For This Useful Post:

    Prometheus (09-20-2017)

  12. #11
    Points: 5,083, Level: 45
    Level completed: 67%, Points required for next Level: 67

    Posts
    109
    Thanks
    19
    Thanked 3 Times in 3 Posts

    Re: OLS vs solving a system of linear equaions

    That's what i needed, thanks. Also learnt the Euclidean norm isn't the same as the Frobenius norm.

    Quick follow up - would it be difficult to constrain the solutions to sum to one?
    Last edited by Prometheus; 09-20-2017 at 07:58 AM.

  13. #12
    Devorador de queso
    Points: 95,819, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,935
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: OLS vs solving a system of linear equaions


    I always wonder what the point of imposing such constraints is. But yes you can do that in software easily enough.
    I don't have emotions and sometimes that makes me very sad.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats