Not Quite Quadratic Regression

[cause]

Hello all,

I predict an u-shaped relation between two variables, y and x. If I perform a quadratic linear regression, i.e. use a model like

y = b1 + b2x + b3x^2 + error,

then the coefficients b2 and b3 are not significant. However, if I change the exponent to something less than 2, e.g. 1.5, I obtain significance. In other words a model like

y = b1 + b2x + b3x^1.5 + error,

yields significant estimates of b2 and b3. The curvature is still quite marked using the exponent of 1.5. I can even use an exponent of 1.1 and obtain significance and a nice shape. But of course I dont think I can simply choose the exponent based on what gives me significance. Or can I?

I have tried to run a non-linear regression where the exponent was a free parameter, but although it tend to yield an exponent around 1 to 2, everything became highly insignificant.

I have also tried to use a splines as well as a piecewise constant formulation, but again the results are less than ideal.

I am wondering if it would be considered bad form to use an exponent of e.g. 1.5 based on the fact that it yields significance? It would appear a little arbitrary in my eyes, since then I could just as well choose an exponent of 1.1.

The non-linearity is rather apparant in a scatterplot (although extremely noisy), and the problem mostly shows up when controlling for other covariates and a simple graphical/nonparametric approach is unfeasible.

I am very interested in suggestions as how to approach this problem. Needless to say, I have been searching high and low for an answer before posting here.

Thank you in advance!

 

[derksheng]

If you pass a specification test (e.g. Ramsey) I don't see how it's different from selecting ^2 "arbitrarily".

I doubt there's anything special about ^2.

But this is an interesting question nonetheless when there's a large interval over the reals that the exponent could assume where you both pass the Ramsey's RESET test and still get significance... Very interesting question.

 

[cause]

Thanks for your reply, derksheng.

You have a good point: ^2 is just as arbitrary as any other reasonable exponent. In that case, my problem becomes one of motivating how I land on e.g. ^1.5 or ^1.1 and why I choose one above the other.

Thank you also for your suggestion of running a RESET test. I will certainly try that!

Further input is welcome!

 

[GretaGarbo]

When I first read this I thought that maybe Aikaike Information Criterion (AIC) could be used to evaluate which model works best. Since there are different number of estimated parameters in the two models (x^2 versus x^p) one need to discriminate between them. And AIC tries to make a trade-off between good-fit and model complexity (the number of parameters).

In a prediction problem cross validation can be used.

When I read derkshengs comment (thanks for that) I came to think of fractional polynomials. To change the title:

“Not quite quadratic regression, but maybe fractional polynomials”.

One extreme is to give the exponent a fixed values, like x^2.

Another extreme is to let the exponent be estimated freely, like x^p.

An in-between case is fractional polynomials.

In fractional polynomial: a model:
FP = b0 + b1*x^p1 + b2*x^p2 + b3*x^p3 +…+ bm*x^pm

Where the p1, p2,… are chosen from a finite set of values like (-2, -1, -0.5, 0, 0.5, 1, 2, 3) (where “x^0” are supposed to mean log(x))

So fractional polynomials are just using a few possible exponents. It becomes a quite simple model.

(I stop here because otherwise someone opens this post and think “it is as long as a thesis” and closes.)

 

[GretaGarbo]

….continued.
How is the model estimated? Eehh, don’t remember. I guess with some iterative procedure to get “good fit”.

(Royston Altman are connected to fractional polynomials)

Fractional polynomials are like a modern variant of the Box-Tisdell from the 1960ies.)

You can estimate fractional polynomials in Stata and I believe so in several packages in R.

Other smoothing possibilities are splines in generalized additive model (gam). In R this can be estimated in the packages gam and mgcv (which stand for m-something general cross validation). (And other smoothing possibilities are like lowess and distance weighted least squares.)
I haven’t really used fractional polynomials myself (but I have use gams).

I haven’t seen it written but I believe someone said to me that fractional polynomials are “about as good” as gam:s. (This is terribly imprecise! Apologize! I also apologize for my even worse than usual English! When I am saying this I am also apologizing if I have typed someone’s name incorrectly.)

 

[cause]

Wow, thank you very much, GretaGarbo! I think the method of fractional polynomials pretty much solves my problem by serving a formal procedure to select the best model with possibly non-integer exponents.

I thank you both for your help! I am sure future "googlers" will also appreciate your responses.

I am still interested in any further input.