+ Reply to Thread
Results 1 to 11 of 11

Thread: MSE RMSE for Modeling/Technique Examination

  1. #1
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    MSE RMSE for Modeling/Technique Examination




    I have heard people talking about creating the best modeling approach, best propensity scores, best approach for dealing with missing data, etc. - and using MSE or RSME to compare the approaches being studied.


    So, MSE = 1/n {sum(Prediction modeli - truthi)^2}, i = observation of approach


    and RMSE = sqrt(MSE)


    I understand you can simulate data to know the true parameters being estimated, but which statistic or value are they actually comparing in "(prediction - truth)" component? Can someone provide possible examples. I am guessing this is straightforward, but I am just unable to conceptualizing it.
    Stop cowardice, ban guns!

  2. #2
    Devorador de queso
    Points: 95,940, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    Code: 
    > set.seed(100)
    > x <- runif(30)
    > y <- 1 + 2*x + rnorm(30)
    > plot(x,y)
    > o <- lm(y ~ x)
    > preds <- predict(o)
    > # To get MSE:
    > mean( (preds - y)^2 )
    [1] 0.7415872
    I don't have emotions and sometimes that makes me very sad.

  3. The Following User Says Thank You to Dason For This Useful Post:

    hlsmith (01-14-2016)

  4. #3
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    Thanks. I know what the MSE is, so you are saying they just average the MSE over a series of varying simulated datasets using the method of interest? Then repeat with the comparative approach?

    If they are simulating the model, how do they not fit exact unbiased predictors? Perhaps they leave out a term or add error?

    I understand why you would simulate data - to know the answer to the null hypothesis, so then they can apply approaches to the simulated data to see which has results closes to the truth and can do this with multiple simulated datasets to test the applicability of the approach within different dataset patterns. But these estimates do not include MSEs unless the outcome is continuous, correct? I guess that is my question as well. I wish I had a published reference to provide an example.
    Stop cowardice, ban guns!

  5. #4
    Devorador de queso
    Points: 95,940, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    I guess I'm still not sure what the question is or what use of MSE you're referring to.
    I don't have emotions and sometimes that makes me very sad.

  6. #5
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    I am sure the issue is related to my lack of having a good example in hand and making too many generalizations.


    If I have time, I am going to try and find an example, below this text is a study that used SuperLearner package to create the 12 best models using V-fold CV, they then weighted the models to create an average MSE for the models and compare these to other approaches. I think I was assuming people used a sample parameter minus the truth. But as Dason noted in the formula, perhaps it is just the predicted observations - Y, aka the truth. So that is how they get the MSEs that they average, by using the MSEs from each study? I believe I want to explore Cross-Validation some more. The study with CV I referenced was:


    Am J Epidemiol. 2013;177(5):443452
    Stop cowardice, ban guns!

  7. #6
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    Currently going to read this article: Am J Epidemiol 2006;163:1149-1156


    The article provides an example of the use of MSE (e.g., examination of propensity scores) that is very comparable to what I mentioned in Post #1, with the outcome not coming from a traditional linear model. Link to paper below:

    http://www.umassmed.edu/uploadedFile..._Selection.pdf

    Last edited by hlsmith; 01-14-2016 at 04:06 PM. Reason: I increase the font size, for better readability
    Stop cowardice, ban guns!

  8. #7
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    Alright, in the study I referenced in post #6, the authors play around with the affects of the following types of variables in the use of propensity scores:
    -variables associated with exposure and outcome (confounder),
    -variables associated solely with exposure, and
    -variables associated solely with outcome

    The purpose of the study was to examine the effects of including these types of variables in the propensity scores of models and estimating exposure effects versus the true exposure effect per simulated dataset.


    In the below equation taken from the article:


    alpha-subscript 4 is the true exposure effect on Y, derived from the simulation (e.g., 0.5); and


    gamma-hat is the estimated exposure effect on Y, which is a log relative risk in this study.




    Can some one help me better understand their approach of determining the squared difference of a log relative risk - true effect. I get why they did everything that they did, I am just getting confused by how they can get an MSE out of a log relative risk, pretty much since it is not a continuous parameter. I guess you can do this, so you can just find differences in effects and create an MSE whenever you want?







    Attached Images  
    Stop cowardice, ban guns!

  9. #8
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    Well there appears to be a bunch of articles out there that use this approach (MSEs in non-traditional linear regression models). The authors typically report bias, MSE, and an absolute error value. I am going to work on finding a good example that actually provides numbers, for say estimated odds ratio - true odds ratio. This way I can have a better understanding of its application beyond just seeing a formula in an article.
    Stop cowardice, ban guns!

  10. #9
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    I believe this concept is now clearer in my mind. Components of the approach that were confusing me were the use of mean square error (MSE) estimates form nonlinear regression models and also, for some reason I was thinking MSE values were bound between 0 and 1.

    My take home points, were that you can apply MSE to any question you have when you want to compare an estimate to the truth. So applications can include comparing new (novel) approaches to older approaches of modeling and when trying to address systematic errors via modeling approaches. The way you are able to do this is by simulating varying datasets and examining estimates versus the known true effects.


    Pretty simple and straightforward idea. Though, it drastically helped when I found published examples.
    Stop cowardice, ban guns!

  11. #10
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination

    This way of preseting different modeling approaches also seems to be able to help address the bias / variance tradeoff in model building.
    Stop cowardice, ban guns!

  12. #11
    Omega Contributor
    Points: 38,413, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,004
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: MSE RMSE for Modeling/Technique Examination


    I think this may also fall under:


    Loss Function
    Stop cowardice, ban guns!

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats