Linear vs. Non-Linear Functions


Less is more. Stay pure. Stay poor.
I am curious on your thoughts on linear vs. nonlinear functions. So I am referring to the projection of the dependent variable into space as a function of covariates. When is it linear vs. non-linear (including a line or in a hyperplane).

In linear regression if there isn't a polynomial and the DV is the sum of the covariates - then it is linear right? And in some time series the terms aren't linear given the DV can be the product of terms. When else is the DV not linear, how about when dealing with logs and other operators.

What is sparking this, is I am fitting a certain algorithm and I feel like it may function better than other main effects algorithms when the DV is a non-linear function of the covariates, but doesn't converge as fast when it is a simple linear function. Thus I am trying to think about settings when relationships would not be linear. Sorry if this post isn't that linear in its point and a little wandering - I am still trying to frame my thoughts, but looking for general input.


Can't make spagetti
This is always a confusion with my students (and quite a few professors), so it makes sense to be confused every now and then. I feel there are a few things that are being conflated in your post that need to be clarified:

(i) A linear function and a linear model are not necessarily the same thing. A linear function has this form:

\(y = mx + b\) for \((m, b)\) some constants, right? However this:

\(y = mx^2 + b\) is not a linear function. It's a quadratic function. HOWEVER, if you turn these into regression equations like:

\(Y = \beta_{0} + \beta_{1}X + \epsilon\)
\(Y = \beta_{0} + \beta_{1}X^{2} + \epsilon\)

They are still linear models. We don't really talk about the "DV being not linear" but I intuit what you mean is the relationship between the DV and IVs is linear. It is of course not a linear relationship in the case of \(Y = \beta_{0} + \beta_{1}X^{2} + \epsilon\) where the relationship is quadratic. But the model itself is linear. The expression I've heard before to describe it is saying it is linear in the parameters. As such, a regression model with first-order interaction effects like:

\(Y = \beta_{0} + \beta_{1}X_{1} + \beta_{2}X_{2} + \beta_{3}X_{1}X_{2} + \epsilon\)

is still a linear model, even if there is a product term. It's a polynomial of degree 2.

(ii) A non-linear model is one where the parameters cannot be expressed in linear terms. Most obvious example for me is:

\(Y = e^{\beta_{0} +\beta_{1}X_{1} + \epsilon}\)

However, notice that this relationship could be linearized such that:

\( log(Y )= \beta_{0} +\beta_{1}X_{1} + \epsilon\)

But that changes the dependent variable from \(Y\) to \(log(Y)\).

Another example which may be closer related to your time series query perhaps would be something like:

\( Y= sin(\beta_{0} +\beta_{1}X_{1} + \epsilon)\)

Because the sine function induces periodicity that is difficult to model with a single line (unless you do piecewise regression).

The way I remember these things is usually thinking that if it has a transcendental function involved (exponentiation, trigonometric functions, etc.) It's a non-linear model. But that does not mean said model cannot be linearized. Logistic regression is perhaps a good example of this, as well as most generalIZED linear models where you try to find a linear solution to a very much non-linear problem.


Less is more. Stay pure. Stay poor.
Well written response. My confusion was not between a linear model and function. I was kind of trying to figure out when a function isn't linear or when a linear model, which in my mind I was calling a type of function, wouldn't be linear in the DV. I know I am kind of confusing things by not quite using the right terminology!


Can't make spagetti
trying to figure out when a function isn't linear or when a linear model, which in my mind I was calling a type of function, wouldn't be linear in the DV.
Well the case of a linear function is easy: a polynomial on any number of variables will be linear as long as the degree of the polynomial is 1*. If the degree is not 1, then the function is no longer linear.

A model is not linear if it isn't linear on the parameters. Hence, all generalized linear models are not linear, say, at first **BUT** they become linearized through the link function (logit, probit, log-log, etc.). It's the same case as with the \(Y = e^{\beta_0 + \beta_1X_1 + \epsilon}\) that implies a non-linear model... BUT nobody really works with that. We all work with \(log(Y)\) because it makes the problem much easier by... you guessed right, linearizing it. If you hang around long enough in this biz you'll realize we almost always try to linearize every problem to make things more interpretable.

*Disclaimer: I am assuming integer-valued exponents for the polynomial. I honestly don't remember how polynomial degrees work with fractional polynomials, because you could have some funky business like x^(1/2)y^(1/2) which is very obviously non-linear
Last edited: