+ Reply to Thread
Results 1 to 14 of 14

Thread: Reviewer questioning my regression in my article, need advice on how/what to respond

  1. #1
    Points: 948, Level: 16
    Level completed: 48%, Points required for next Level: 52

    Posts
    7
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Reviewer questioning my regression in my article, need advice on how/what to respond




    Hi guys. I’ve submitted my first article and a reviewer is questioning my stats. I never had this problem before and I need your help!

    Quick rundown: it is a clinical study (n=150ish) and I was interested in how well exposure to a certain toxin (main predictor) could predict scores on a clinical questionnaire (DV). I used a standard linear regression model (which means the "enter" method) and included other predictors that are known to affect scores on my DV (age, education, disease duration and motor disability). All variables are on a continuous scale. I’ve found that my model is significant (R2 = .285), and my main predictor (toxin exposure) is also significant (Standardized Beta = -.195), along with two demographic predictor (age = -.233* and education = .334*). There is no collinearity problem as per the VIF index.

    One reviewer is questioning my model because he says that since about 70 people out of 150 had an exposure value of 0 (non exposed), my model is only a fit for to those who were exposed to the toxin (about 80). My understanding of the regression model and the number of degree of freedom in the ANOVA table makes me thinks he is wrong. For instance, if I were to run a simple Pearson correlation between toxin exposure and my DV, I would use the whole sampel and not exclude those who were not exposed. Can you please help me figure this out and do you have any advice on how to tell the reviewer/editor about that?

    He is also criticizing the fact that in the model I report in the article as a table includes both significant and non-significant predictors. I’ve seen countless articles reporting it that way, i.e. the regression as it was ran in SPSS. He also asks me what were my criteria to retain or add variables to the model, but I’ve used an “enter” method were all the predictors are entered at the same time and the significant predictors are only the marginal effects (unique individual effect) of each on the DV.

    So please, can you confirm that I am right/wrong, and do you have any advice on how to write this to the editor/reviewer so my paper doesn’t get rejected in the second round?

    THANK YOU SO MUCH!

    PS: attached histogram and normality plot of the residuals
    /RESIDUALS HISTOGRAM(ZRESID) NORMPROB(ZRESID).
    Last edited by nightale; 07-23-2015 at 01:01 AM.

  2. #2
    Omega Contributor
    Points: 38,396, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,001
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    All and all, given you description - I would say you are fine. If the non-significant variables in the model are of clinical interest, you are fine leaving them in the model. I as well have seen and done this. If you report an R^2 just make sure it is an adjusted measure, so the extra terms don't inflate its value.


    Now to the big question, Toxin?? I think you are fine as long as the residual aren't compromised, which they seem pretty good. Now whether you keep the null values in the model just comes down to your question. Are you trying to generalize the results to other comparable samples or to just those exposed? In my opinion you seem fine. Depending on the values of the Toxin and its dose-effect on the outcome, you could also run another model with it as a categorical variable (e.g., 0, moderate, high). Reporting both ways may help stave off some of their criticisms.
    Stop cowardice, ban guns!

  3. #3
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    One reviewer is questioning my model because he says that since about 70 people out of 150 had an exposure value of 0 (non exposed), my model is only a fit for to those who were exposed to the toxin (about 80).
    I don't really understand this comment. I assume from your comment that you are coding this as a dummy variable. That is 1 exponsed to toxin and zero not. If so it would make no sense to exclude those who are not subject to the toxin, how would you show that impact of the toxin existed if you only include cases where they were exposed and did not include those that were not? I am guessing I simply don't understand what you are saying here... If you are coding it as a dosage (that is an interval variable) then I don't understand why you would exclude the 0 level, but your field may be different. It depends on your theory here. If you think that all that matters is if you are exposed or not than using this as a dummy variable would make sense and (I think) would address the concern raised. If you think the impact goes up with level then you should leave it as a continuous variable.
    Personally I do not understand the logic of your reviewer that you should exclude those who are not subject to the effect if you feel the effect has an impact. In that case you would expect those with no exposure to show a different score than those who did and your test result should certainly not exclude them. I have never seen it suggested that you do. This has nothing to do with generalizability and if you applied the logic they are using then dummy variables that modeled intervention [this did occur/did not] such as receiving a medicine would not be valid. It would be like arguing that test of a given treatment were only valid for those who received the treatment and not to those that did not.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  4. The Following User Says Thank You to noetsi For This Useful Post:

    nightale (07-17-2015)

  5. #4
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    Incidently I think one way of addressing the comments made is to find articles which do the same thing you did and cite them. For instance one of the many articles that show variables that are/are not signficant in tables. The logic of showing signficant and non-signficant variables is a simple one. The point of the analysis is to show what mattered and what did not. Only including those variables that are signficant makes little sense then because it defeats the whole point of the analysis. Various authors agree that leaving in variables that make theoretical sense even that are not signficant is fine. You should find one or two and cite them.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  6. #5
    Omega Contributor
    Points: 38,396, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,001
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    I see the benefit of include and not including the non-significant variables. This is as long as you not they were non-significant and you kept them out. Though, you can most definitely keep them in. This almost falls into a weird cousin of publication bias. Keep em' in if you want


    Noetsi, It is my interpretation that the OP included toxin as a continuous variable, though may patients did not have a detectable level, so 80 had a value of 0. Which excluding these patients seems crazy, because how else to you evaluation risk in unexposed groups. Perhaps drawing a causal diagram may help with the paper. You would have two endogenous exposure variables going into the outcome, those exposed and not exposed. If anything it may help you develop the idea in your own mind.


    Still think you are fine, you just need to articulate your rationale back to the reviewer. Its a process and many times you can get reviewer's with differing comments and justifications.
    Stop cowardice, ban guns!

  7. The Following User Says Thank You to hlsmith For This Useful Post:

    nightale (07-17-2015)

  8. #6
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    Noetsi, It is my interpretation that the OP included toxin as a continuous variable, though may patients did not have a detectable level, so 80 had a value of 0. Which excluding these patients seems crazy, because how else to you evaluation risk in unexposed groups.
    I agree that it makes absolutely no sense to exclude those that did not experience the event in an analysis specifically targeted at measuring the impact of that event So much so that I think the reviewer must have meant something else or just got confused. This is true whether the IV is measured as a dummy or as a continuous variable.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  9. #7
    Points: 948, Level: 16
    Level completed: 48%, Points required for next Level: 52

    Posts
    7
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    First of all, thank you so much everyone for your help.

    Depending on the values of the Toxin and its dose-effect on the outcome, you could also run another model with it as a categorical variable (e.g., 0, moderate, high). Reporting both ways may help stave off some of their criticisms.
    My "toxin" (it is not really a toxin, but I don't want to give too hints so people can identify it) is a continuous variable, for instance it could conceptualized be the same as number of coffees per week on average. I also did compare those exposed with those not exposed with a simple ANCOVA and did find a significant difference between the groups, so I guess it confirms what you are saying.

    Personally I do not understand the logic of your reviewer that you should exclude those who are not subject to the effect if you feel the effect has an impact. In that case you would expect those with no exposure to show a different score than those who did and your test result should certainly not exclude them. I have never seen it suggested that you do. This has nothing to do with generalizability and if you applied the logic they are using then dummy variables that modeled intervention [this did occur/did not] such as receiving a medicine would not be valid. It would be like arguing that test of a given treatment were only valid for those who received the treatment and not to those that did not.
    Incidently I think one way of addressing the comments made is to find articles which do the same thing you did and cite them. For instance one of the many articles that show variables that are/are not signficant in tables. The logic of showing signficant and non-signficant variables is a simple one.
    I agree with you two, thank you, this will help me write back a better response. I checked for a second opinion with other people and they all agree the reviewers must be confused on how linear regression works.

    I think you are fine as long as the residual aren't compromised, which they seem pretty good.
    TBH, that’s the part I understand the less. I mean everything looks good on those graph, but what am I supposed to check for exactly? (I did only a “visual expection”). Also, I added a scatterplot of zred over zpred, if you want to check that there is no problem.

    Thank you everyone! That helped me a lot!
    Last edited by nightale; 07-23-2015 at 01:01 AM.

  10. #8
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    I checked for a second opinion with other people and they all agree the reviewers must be confused on how linear regression works.
    thats a scary statement in that linear regression is by far the simplest form of regression and you are talking I think about medical or psychological research. But I have no doubt it occurs. Articles on logistic regression, [which is more complex than linear regression IMHO] have found elite medical journals such the New England Journal of Medicine and JAMA making really basic mistakes in logistic regression that signficantly changes the results or
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  11. #9
    Omega Contributor
    Points: 38,396, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,001
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    Yeah, everything looks fine. Have you tried running it without those three potential outliers?
    Stop cowardice, ban guns!

  12. #10
    Points: 948, Level: 16
    Level completed: 48%, Points required for next Level: 52

    Posts
    7
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    Quote Originally Posted by noetsi View Post
    thats a scary statement in that linear regression is by far the simplest form of regression and you are talking I think about medical or psychological research. But I have no doubt it occurs. Articles on logistic regression, [which is more complex than linear regression IMHO] have found elite medical journals such the New England Journal of Medicine and JAMA making really basic mistakes in logistic regression that signficantly changes the results or
    I agree! I don't know if you've read "The (mis)reporting of statistical results in psychology journals" by Bakker and Wicherts in 2011, it is rather unflattering for the field.

    Quote Originally Posted by hlsmith View Post
    Yeah, everything looks fine. Have you tried running it without those three potential outliers?
    Actually I realized that this is not the scatterplot of the "final" iteration, because they were removed from the analysis as the computed Mahalanobis distance was over the threshold. I must have loaded the earlier database file when I got that scatterplot. EDIT: I forgot to add the "case" filter when I loaded the scatterplot.

    Well, thank you so much guys, you are amazing. I'm always impressed that some people are kind enough to willingly give some of their own time like that!

  13. #11
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    We plan to steal your ideas and write articles on them [you might laugh at that idea, but we have a collegue here who experienced this in reality not on this site which is awesome, but in his research. That is his ideas were plagarised by a collegue].

    If you find a good way to predict commodity market I would be happy to review your article on that
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  14. #12
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    hi,
    just wondering: including a lot of the non-exposed patients will simply increase the error term in the anova analysis and thus make the tests for significance a lot less powerful, I guess. So, this might just not be a good idea. What do you guys think?

    regards

  15. #13
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp

    But you are testing if an intervention worked or not. How can you test that if you only include interventions and none of the non-interventions. It would be like doing a t test with only one level
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  16. #14
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Reviewer questioning my regression in my article, need advice on how/what to resp


    I suppose we test whether the coefficient a1 is different from zero in the equation Y=a0 +a1X. Adding a lot of points to the X=0 group is not likely to bring much information about a1 - I think this is different from a simple test of whether X has an effect or not, in which case you would be absolutely right.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats