Code:> set.seed(100) > x <- runif(30) > y <- 1 + 2*x + rnorm(30) > plot(x,y) > o <- lm(y ~ x) > preds <- predict(o) > # To get MSE: > mean( (preds - y)^2 ) [1] 0.7415872
I have heard people talking about creating the best modeling approach, best propensity scores, best approach for dealing with missing data, etc. - and using MSE or RSME to compare the approaches being studied.
So, MSE = 1/n {sum(Prediction modeli - truthi)^2}, i = observation of approach
and RMSE = sqrt(MSE)
I understand you can simulate data to know the true parameters being estimated, but which statistic or value are they actually comparing in "(prediction - truth)" component? Can someone provide possible examples. I am guessing this is straightforward, but I am just unable to conceptualizing it.
Stop cowardice, ban guns!
Code:> set.seed(100) > x <- runif(30) > y <- 1 + 2*x + rnorm(30) > plot(x,y) > o <- lm(y ~ x) > preds <- predict(o) > # To get MSE: > mean( (preds - y)^2 ) [1] 0.7415872
I don't have emotions and sometimes that makes me very sad.
hlsmith (01-14-2016)
Thanks. I know what the MSE is, so you are saying they just average the MSE over a series of varying simulated datasets using the method of interest? Then repeat with the comparative approach?
If they are simulating the model, how do they not fit exact unbiased predictors? Perhaps they leave out a term or add error?
I understand why you would simulate data - to know the answer to the null hypothesis, so then they can apply approaches to the simulated data to see which has results closes to the truth and can do this with multiple simulated datasets to test the applicability of the approach within different dataset patterns. But these estimates do not include MSEs unless the outcome is continuous, correct? I guess that is my question as well. I wish I had a published reference to provide an example.
Stop cowardice, ban guns!
I guess I'm still not sure what the question is or what use of MSE you're referring to.
I don't have emotions and sometimes that makes me very sad.
I am sure the issue is related to my lack of having a good example in hand and making too many generalizations.
If I have time, I am going to try and find an example, below this text is a study that used SuperLearner package to create the 12 best models using V-fold CV, they then weighted the models to create an average MSE for the models and compare these to other approaches. I think I was assuming people used a sample parameter minus the truth. But as Dason noted in the formula, perhaps it is just the predicted observations - Y, aka the truth. So that is how they get the MSEs that they average, by using the MSEs from each study? I believe I want to explore Cross-Validation some more. The study with CV I referenced was:
Am J Epidemiol. 2013;177(5):443–452
Stop cowardice, ban guns!
Currently going to read this article: Am J Epidemiol 2006;163:1149-1156
The article provides an example of the use of MSE (e.g., examination of propensity scores) that is very comparable to what I mentioned in Post #1, with the outcome not coming from a traditional linear model. Link to paper below:
http://www.umassmed.edu/uploadedFile..._Selection.pdf
Last edited by hlsmith; 01-14-2016 at 04:06 PM. Reason: I increase the font size, for better readability
Stop cowardice, ban guns!
Alright, in the study I referenced in post #6, the authors play around with the affects of the following types of variables in the use of propensity scores:
-variables associated with exposure and outcome (confounder),
-variables associated solely with exposure, and
-variables associated solely with outcome
The purpose of the study was to examine the effects of including these types of variables in the propensity scores of models and estimating exposure effects versus the true exposure effect per simulated dataset.
In the below equation taken from the article:
alpha-subscript 4 is the true exposure effect on Y, derived from the simulation (e.g., 0.5); and
gamma-hat is the estimated exposure effect on Y, which is a log relative risk in this study.
Can some one help me better understand their approach of determining the squared difference of a log relative risk - true effect. I get why they did everything that they did, I am just getting confused by how they can get an MSE out of a log relative risk, pretty much since it is not a continuous parameter. I guess you can do this, so you can just find differences in effects and create an MSE whenever you want?
Stop cowardice, ban guns!
Well there appears to be a bunch of articles out there that use this approach (MSEs in non-traditional linear regression models). The authors typically report bias, MSE, and an absolute error value. I am going to work on finding a good example that actually provides numbers, for say estimated odds ratio - true odds ratio. This way I can have a better understanding of its application beyond just seeing a formula in an article.
Stop cowardice, ban guns!
I believe this concept is now clearer in my mind. Components of the approach that were confusing me were the use of mean square error (MSE) estimates form nonlinear regression models and also, for some reason I was thinking MSE values were bound between 0 and 1.
My take home points, were that you can apply MSE to any question you have when you want to compare an estimate to the truth. So applications can include comparing new (novel) approaches to older approaches of modeling and when trying to address systematic errors via modeling approaches. The way you are able to do this is by simulating varying datasets and examining estimates versus the known true effects.
Pretty simple and straightforward idea. Though, it drastically helped when I found published examples.
Stop cowardice, ban guns!
This way of preseting different modeling approaches also seems to be able to help address the bias / variance tradeoff in model building.
Stop cowardice, ban guns!
I think this may also fall under:
Loss Function
Stop cowardice, ban guns!
Tweet |