MSE RMSE for Modeling/Technique Examination


Omega Contributor
I have heard people talking about creating the best modeling approach, best propensity scores, best approach for dealing with missing data, etc. - and using MSE or RSME to compare the approaches being studied.

So, MSE = 1/n {sum(Prediction modeli - truthi)^2}, i = observation of approach

and RMSE = sqrt(MSE)

I understand you can simulate data to know the true parameters being estimated, but which statistic or value are they actually comparing in "(prediction - truth)" component? Can someone provide possible examples. I am guessing this is straightforward, but I am just unable to conceptualizing it.


Ambassador to the humans
> set.seed(100)
> x <- runif(30)
> y <- 1 + 2*x + rnorm(30)
> plot(x,y)
> o <- lm(y ~ x)
> preds <- predict(o)
> # To get MSE:
> mean( (preds - y)^2 )
[1] 0.7415872


Omega Contributor
Thanks. I know what the MSE is, so you are saying they just average the MSE over a series of varying simulated datasets using the method of interest? Then repeat with the comparative approach?

If they are simulating the model, how do they not fit exact unbiased predictors? Perhaps they leave out a term or add error?

I understand why you would simulate data - to know the answer to the null hypothesis, so then they can apply approaches to the simulated data to see which has results closes to the truth and can do this with multiple simulated datasets to test the applicability of the approach within different dataset patterns. But these estimates do not include MSEs unless the outcome is continuous, correct? I guess that is my question as well. I wish I had a published reference to provide an example.


Omega Contributor
I am sure the issue is related to my lack of having a good example in hand and making too many generalizations.

If I have time, I am going to try and find an example, below this text is a study that used SuperLearner package to create the 12 best models using V-fold CV, they then weighted the models to create an average MSE for the models and compare these to other approaches. I think I was assuming people used a sample parameter minus the truth. But as Dason noted in the formula, perhaps it is just the predicted observations - Y, aka the truth. So that is how they get the MSEs that they average, by using the MSEs from each study? I believe I want to explore Cross-Validation some more. The study with CV I referenced was:

[FONT=AdvOTeeb42827.I][FONT=AdvOTeeb42827.I]Am J Epidemiol. [/FONT][/FONT]2013;177(5):443[FONT=AdvTT5843c571+20][FONT=AdvTT5843c571+20][/FONT][/FONT]452


Omega Contributor
Alright, in the study I referenced in post #6, the authors play around with the affects of the following types of variables in the use of propensity scores:
-variables associated with exposure and outcome (confounder),
-variables associated solely with exposure, and
-variables associated solely with outcome

The purpose of the study was to examine the effects of including these types of variables in the propensity scores of models and estimating exposure effects versus the true exposure effect per simulated dataset.

In the below equation taken from the article:

alpha-subscript 4 is the true exposure effect on Y, derived from the simulation (e.g., 0.5); and

gamma-hat is the estimated exposure effect on Y, which is a log relative risk in this study.

Can some one help me better understand their approach of determining the squared difference of a log relative risk - true effect. I get why they did everything that they did, I am just getting confused by how they can get an MSE out of a log relative risk, pretty much since it is not a continuous parameter. I guess you can do this, so you can just find differences in effects and create an MSE whenever you want?



Omega Contributor
Well there appears to be a bunch of articles out there that use this approach (MSEs in non-traditional linear regression models). The authors typically report bias, MSE, and an absolute error value. I am going to work on finding a good example that actually provides numbers, for say estimated odds ratio - true odds ratio. This way I can have a better understanding of its application beyond just seeing a formula in an article.


Omega Contributor
I believe this concept is now clearer in my mind. Components of the approach that were confusing me were the use of mean square error (MSE) estimates form nonlinear regression models and also, for some reason I was thinking MSE values were bound between 0 and 1.

My take home points, were that you can apply MSE to any question you have when you want to compare an estimate to the truth. So applications can include comparing new (novel) approaches to older approaches of modeling and when trying to address systematic errors via modeling approaches. The way you are able to do this is by simulating varying datasets and examining estimates versus the known true effects.

Pretty simple and straightforward idea. Though, it drastically helped when I found published examples.


Omega Contributor
This way of preseting different modeling approaches also seems to be able to help address the bias / variance tradeoff in model building.