backwards stepwise multiple regression, collinearity. Rejected paper.

#1
Hi helpful people!

My research paper was recently rejected and some of the feedback I received was in relation to the statistical tests done/not done. I would like help in clarifying what I could do differently as the feedback was not to informative.
I am attempting to see which baseline characteristics (my independent variables) can predict who will improve the most in my dependent variable after an intervention. As it is not published yet I won’t give to many details but a similar example would be trying to decide if any baseline characteristics in humans (such as muscle mass, age, gender, alcohol use, pulse rate etc) can predict improvement in 100m foot race times after undergoing a strength exercise program. I have a cohort of about 100 individuals all undergoing the same intervention.
In order to test this I collected data on all my baseline values and measured participants 100m times before and after undergoing the strength exercise program. I then made a multiple regression model were I included previous known confounders and my baseline characteristics of interest and used backwards stepwise removal of non-significant regressors to end up with a model of 3 independent variables significantly associating with improvement in 100m race times. For the sake of the argument let’s make up the following; gender, thigh muscle mass and smoking status (yes/no).

I was asked/critiqued on the following (again examples are made up);

1; type of sports shoe is a well-known determinant of 100 m race times, were improvement in race times still associated with baseline thigh muscle mass after adjusting for choice of sport shoe,?

-type of sport shoe was one of the independent variables included in my multiple regression model, however it was not significant when included with the other independent variables so it was removed in the backward stepwise removal process. Is any other statistical test more appropriate to run?

2, Could collinearity explain the results as several of the independent variables are likely to be similar

-I ran collinearity diagnostics in SPSS and did not receive any VIF values over 4 (with only one independent variable had a VIF at 4, the rest were under 3)

3; discuss regression to the mean as an explanation to my results

-I concede that it is likely that regression to the mean plays a part in which individuals improved the most/least but I don’t see how this impacts on the baseline characteristics in a significant way other than that these individuals are given greater weight in the results since they show the biggest change. I divided my cohort into tertiles based on improvement in race times and did not find that they differed in baseline values in any of my independent variables of interest.

Any input much appreciated!
 

hlsmith

Omega Contributor
#2
What exactly was outcome (time, percent change, etc.)?

Did you have initial time in the model?

I believe it may be JAMA instructions to authors that has an excellent description of modelling before after change. I will look up the source tomorrow.

How many independent tests did you initially have in the model?

We will tease things out, but perhaps some of your issues may be related to your descriptive write-ups. Did they give you a flat out rejection or major/minor revisions?

A control group would have been nice, and perhaps still doable!
 

CowboyBear

Super Moderator
#3
As it is not published yet I won’t give to many details but a similar example would be trying to decide if any baseline characteristics in humans (such as muscle mass, age, gender, alcohol use, pulse rate etc) can predict improvement in 100m foot race times after undergoing a strength exercise program.
I think it might be better if you just told us what the actual research is about. Many researchers nowadays actively put up preprints of their work long before publication or acceptance - empirical research isn't Game of Thrones, putting up spoilers doesn't hurt anyone :). Perhaps if we knew more about what was actually going on we might be able to give more informative feedback?

I included previous known confounders and my baseline characteristics of interest and used backwards stepwise removal of non-significant regressors to end up with a model of 3 independent variables significantly associating with improvement in 100m race times.
Stepwise regression is pretty universally seen as a bad idea these days; it results in biased parameter estimates. In your case in particular it will probably result in you excluding confounding variables from your model that should actually be included (but that happen not to be statistically significant due to your fairly small sample size). If I was reviewing the paper I would want you to just include known confounders and forget the stepwise process.
 
#4
thank you for your replies.

I had 12 IVs enter the model, all the well established confounders in the field (8) and my 4 IVs of interest. Outcome was change in time and I did not have initial time in the model, however neither of the reviewers had any comment about this (unless that is what the regression to the mean was about, however he/she did not word it that way). I was debating if I should include it or not.

I used stepwise backwards as I felt it was way to many IVs to have reliable model. But I do see your point as how it may exclude an important confounder. What is a better way, just pick the most important confounders and 1-2 new IVs of interest and enter into a multiple regression model without any stepwise selection?
 

CowboyBear

Super Moderator
#5
What is a better way, just pick the most important confounders and 1-2 new IVs of interest and enter into a multiple regression model without any stepwise selection?
12 predictors in a model with 100 observations is a bit complex, but not unreasonably so. You can just include them all. Certainly don't now try to decide which IVs to include after you've already seen the results.
 

ondansetron

TS Contributor
#6
If you're not directly interested in the 8 that are believed to already be important, you could try "combining" those 8 via principal components analysis to reduce the dimensions youi're dealing with. You could include the prinicpal components in the model and focus on the 4 new variables.
 
#8
thanks for all the input so far! Will sink my teeth into the refrence you provided tonight but at a quick glance most of the criteria for calculating change from baseline actually seem ok in my study.

I now made one model with all independent IVs and I have another question. I am required to report the Rsquared value of the model which in this case is rather high at .65, however many of the IVs in the model are not significant (p-values at .8 and .6 etc). Is this a problem? I feel that the Raquared value in this case is not very informative with such a model.
 

hlsmith

Omega Contributor
#9
Building a model is a process. Not everyone is going to agree on approaches. Ideally you write out exactly what you plan to do, before you even begin. Thus "investigator degrees of freedom" don't come into play. Such actions come with experience. Point in hand, unless some of the confounders are really cemented into the literature, I am not sure I would keep the 0.8 variable around. Options include removing the variable and seeing if any other coefficients (suspected relationships) change.


There are adjusted r^2 measures, which control for the number of variables in the model. The traditional r^2 will become inflated when more variables are added to the model, thus adjusted r^2 is more appropriate.
 
#10
thanks for all the input so far! Will sink my teeth into the refrence you provided tonight but at a quick glance most of the criteria for calculating change from baseline actually seem ok in my study.

I now made one model with all independent IVs and I have another question. I am required to report the Rsquared value of the model which in this case is rather high at .65, however many of the IVs in the model are not significant (p-values at .8 and .6 etc). Is this a problem? I feel that the Raquared value in this case is not very informative with such a model.
You should (or could) ignore p-values for variables that are "well-established" by the prior literature or theory. It doesn't make good sense to, nor is it judicious to, look at p-values (or CIs) for "significance" of variables that are "known" in advance to be "important".
 

CowboyBear

Super Moderator
#11
If you're not directly interested in the 8 that are believed to already be important, you could try "combining" those 8 via principal components analysis to reduce the dimensions youi're dealing with. You could include the prinicpal components in the model and focus on the 4 new variables.
True, but keep in mind that if you do this then the model doesn't actually include controls for the 8 potential confounding variables - it just controls for a smaller subset of principal components based on these original variables. Jake (one of our forum regulars) recently co-wrote an article discussing how "controlling" for potential confounds that you haven't measured perfectly results in quite serious problems: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719
 
#12
True, but keep in mind that if you do this then the model doesn't actually include controls for the 8 potential confounding variables - it just controls for a smaller subset of principal components based on these original variables. Jake (one of our forum regulars) recently co-wrote an article discussing how "controlling" for potential confounds that you haven't measured perfectly results in quite serious problems: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719
Yes, that's worth noting, as the PCA will lose some of the information in the underlying variables when they're combined. I was thinking that if the OP believed the "confounders" to be highly similar variables, it might be worth reducing the dimensions to capture as much of that information as possible. As always, approaching the research question with a few different techniques and validation methods will be helpful to determine stability of the conclusions. I'll have to read the article, now!
 

hlsmith

Omega Contributor
#13
odan's suggestion reminded me of the use of a propensity score term (kind of). However there is not one variable of interest and I am unfamiliar enough with PCA to know if you have multiple endogeneity, perhaps unfaithfulness, etc., what ramifications can occur.

Comes down to also drawing a causal diagram illustrating relationships and signs!!
 
#14
I have another question. After reading your excellent reference on change from baseline I followed some of the links provided there and ended up at

http://biostat.mc.vanderbilt.edu/wiki/Main/MeasureChange

They recommend many things to ensure you are chosing an ok effect measure of change among them;

"Plot difference in pre and post values vs. the average of the pre and post values (Bland-Altman plot). If this shows no trend, the simple differences are adequate summaries of the effects, i.e., they are independent of initial measurements."

I am only familliar with the Bland-Altman plot when it comes to validating that two techniques measure the same thing, here it is used in "reverse". I tried this with my values and did not get a significant association indicating that the measures are more likely independent from each other. Is this a common technique to use? Should I report this in my paper as some proof that regression to the mean is limited in my study (as well as the other criteria listed in the first link)

Cheers for any input