Linear vs. Non-Linear Functions

hlsmith

Less is more. Stay pure. Stay poor.
#1
I am curious on your thoughts on linear vs. nonlinear functions. So I am referring to the projection of the dependent variable into space as a function of covariates. When is it linear vs. non-linear (including a line or in a hyperplane).

In linear regression if there isn't a polynomial and the DV is the sum of the covariates - then it is linear right? And in some time series the terms aren't linear given the DV can be the product of terms. When else is the DV not linear, how about when dealing with logs and other operators.

What is sparking this, is I am fitting a certain algorithm and I feel like it may function better than other main effects algorithms when the DV is a non-linear function of the covariates, but doesn't converge as fast when it is a simple linear function. Thus I am trying to think about settings when relationships would not be linear. Sorry if this post isn't that linear in its point and a little wandering - I am still trying to frame my thoughts, but looking for general input.
 

spunky

Can't make spagetti
#2
This is always a confusion with my students (and quite a few professors), so it makes sense to be confused every now and then. I feel there are a few things that are being conflated in your post that need to be clarified:

(i) A linear function and a linear model are not necessarily the same thing. A linear function has this form:

\(y = mx + b\) for \((m, b)\) some constants, right? However this:

\(y = mx^2 + b\) is not a linear function. It's a quadratic function. HOWEVER, if you turn these into regression equations like:

\(Y = \beta_{0} + \beta_{1}X + \epsilon\)
\(Y = \beta_{0} + \beta_{1}X^{2} + \epsilon\)

They are still linear models. We don't really talk about the "DV being not linear" but I intuit what you mean is the relationship between the DV and IVs is linear. It is of course not a linear relationship in the case of \(Y = \beta_{0} + \beta_{1}X^{2} + \epsilon\) where the relationship is quadratic. But the model itself is linear. The expression I've heard before to describe it is saying it is linear in the parameters. As such, a regression model with first-order interaction effects like:

\(Y = \beta_{0} + \beta_{1}X_{1} + \beta_{2}X_{2} + \beta_{3}X_{1}X_{2} + \epsilon\)

is still a linear model, even if there is a product term. It's a polynomial of degree 2.

(ii) A non-linear model is one where the parameters cannot be expressed in linear terms. Most obvious example for me is:

\(Y = e^{\beta_{0} +\beta_{1}X_{1} + \epsilon}\)

However, notice that this relationship could be linearized such that:

\( log(Y )= \beta_{0} +\beta_{1}X_{1} + \epsilon\)

But that changes the dependent variable from \(Y\) to \(log(Y)\).

Another example which may be closer related to your time series query perhaps would be something like:

\( Y= sin(\beta_{0} +\beta_{1}X_{1} + \epsilon)\)

Because the sine function induces periodicity that is difficult to model with a single line (unless you do piecewise regression).

The way I remember these things is usually thinking that if it has a transcendental function involved (exponentiation, trigonometric functions, etc.) It's a non-linear model. But that does not mean said model cannot be linearized. Logistic regression is perhaps a good example of this, as well as most generalIZED linear models where you try to find a linear solution to a very much non-linear problem.
 

hlsmith

Less is more. Stay pure. Stay poor.
#3
Well written response. My confusion was not between a linear model and function. I was kind of trying to figure out when a function isn't linear or when a linear model, which in my mind I was calling a type of function, wouldn't be linear in the DV. I know I am kind of confusing things by not quite using the right terminology!
 

spunky

Can't make spagetti
#4
trying to figure out when a function isn't linear or when a linear model, which in my mind I was calling a type of function, wouldn't be linear in the DV.
Well the case of a linear function is easy: a polynomial on any number of variables will be linear as long as the degree of the polynomial is 1*. If the degree is not 1, then the function is no longer linear.

A model is not linear if it isn't linear on the parameters. Hence, all generalized linear models are not linear, say, at first **BUT** they become linearized through the link function (logit, probit, log-log, etc.). It's the same case as with the \(Y = e^{\beta_0 + \beta_1X_1 + \epsilon}\) that implies a non-linear model... BUT nobody really works with that. We all work with \(log(Y)\) because it makes the problem much easier by... you guessed right, linearizing it. If you hang around long enough in this biz you'll realize we almost always try to linearize every problem to make things more interpretable.




*Disclaimer: I am assuming integer-valued exponents for the polynomial. I honestly don't remember how polynomial degrees work with fractional polynomials, because you could have some funky business like x^(1/2)y^(1/2) which is very obviously non-linear
 
Last edited:

hlsmith

Less is more. Stay pure. Stay poor.
#5
@spunky et al.

I am working on this project again and I was wondering if someone could whip up a small non-linear relationship via a data generating function for me. I have ran multiple functions and wanted to run one non-linear function that someone else created - so I can remove the researcher degree of freedom and bias related to me saying that the model worked well on a function that "I" created. So the simulation would have a dependent variable and a couple of independent variables and could even include a non-constant
E.g., of other functions that I have ran so far include:

BMI = wt/ht**2
plasma osmolality
X1**2 - X2**2 + X2 - 1
(Z2 + Z4 + 20)**2
etc.

You can provide it in R code or just list it out and I can recreate it.
 
#6
If you hang around long enough in this biz you'll realize we almost always try to linearize every problem to make things more interpretable.
Sometimes I do studies of video games as a player looking on a semi-opaque system of inter-related variables. What I've found in that domain is if I see data that approximates an acceleration, then my first guess is the x^2 component might not be a literal x * x and instead be a composite of two linear variables x1 * x2
 
Last edited: