Removing terms from a model

noetsi

No cake for spunky
#1
This comes from Harrell's book on regression. I am not sure I understand what he is saying (well that happens a lot). :p

"From the above discussion a general principle emerges. Whenever the response variable is informally or formally linked, in an unmasked fashion, to particular parameters that may be deleted from the model, special adjustments must be made in P-values, standard errors, test statistics, and confidence limits, in order for these statistics to have the correct interpretation. Examples of strategies that are improper without special adjustments (e.g., using the bootstrap) include examining a frequency table or scatterplot to decide that an association is too weak for the predictor to be included in the model at all or to decide that the relationship appears so linear that all nonlinear terms should be omitted. It is also valuable to consider the reverse situation; that is, one posits a simple model and then additional analysis or outside subject matter information makes the analyst want to generalize the model."

I think this means if you specify a model and then remove parts of it, including non-linear terms, the p values and CI are no longer valid. I am not sure if they can be made valid again.

Does this effect the effect sizes as well?
 

noetsi

No cake for spunky
#2
I decided to ask all the questions from the Harrell book in one place :p

"As stated in the Preface, the strategy emphasized in this text, stemming from the last philosophy, is to decide how many degrees of freedom can be “spent,” where they should be spent, and then to spend them. If statistical tests or confidence limits are required, later reconsideration of how d.f. are spent is not usually recommended."

How do you spend degrees of freedom? I think he means things like deciding if the effect should be linear or not. But this seems like a strange way to say that.
 

Dason

Ambassador to the humans
#3
It's just a way of saying including additional terms in the model (whether linear terms for other predictors, polynomial terms for already included variables, or interaction terms). Including any of those will "spend" degrees of freedom.
 

noetsi

No cake for spunky
#4
I don't understand how you can generate one p value this way for a set of variables.

"8. When you can test for model complexity in a very structured way, you may be able to simplify the model without a great need to penalize the final model for having made this initial look. For example, it can be advisable to test an entire group of variables (e.g., those more expensive to collect) and to either delete or retain the entire group for further modeling, based on a single P-value (especially if the P value is not between 0.05 and 0.2)."
 

noetsi

No cake for spunky
#5
It's just a way of saying including additional terms in the model (whether linear terms for other predictors, polynomial terms for already included variables, or interaction terms). Including any of those will "spend" degrees of freedom.
That is what I guessed. I have never seen that terminology before. I don't think anyone can assign degrees of freedom :p
 

Dason

Ambassador to the humans
#6
I don't understand how you can generate one p value this way for a set of variables.

"8. When you can test for model complexity in a very structured way, you may be able to simplify the model without a great need to penalize the final model for having made this initial look. For example, it can be advisable to test an entire group of variables (e.g., those more expensive to collect) and to either delete or retain the entire group for further modeling, based on a single P-value (especially if the P value is not between 0.05 and 0.2)."
Look up full versus reduced tests. It's the same math that all of the other tests are built on.
 

noetsi

No cake for spunky
#7
So it is a chi squared test against two models one nested in the other?

Part of my problem is that Harrell uses very different wording than I am used to.