+ Reply to Thread
Results 1 to 10 of 10

Thread: New normed Goodness of Fit Measures ?

  1. #1
    Points: 7,821, Level: 59
    Level completed: 36%, Points required for next Level: 129

    Posts
    159
    Thanks
    1
    Thanked 7 Times in 7 Posts

    New normed Goodness of Fit Measures ?




    Hello everybody!

    I am currently working on a robust regression problem, where Model R^2 and Spearman correlation recommend different models.

    Are there new normed Goodness of Fit-/ Association-/ Correlation Measures away from the ones mentioned before (scale of target: interval)? With normed I am referring to at least a normed maximum fit (like 1 in R^2 and Correlation).

    Best case they will also work for robust problems.

    Thanks
    Consuli
    Prediction is very difficult, especially about the future. (Niels Bohr)

  2. #2
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: New normed Goodness of Fit Measures ?

    Hmm, can you provide more details!


    Would a grouping of predictions (e.g., by decile) vs. observed values be of any benefit?
    Stop cowardice, ban guns!

  3. #3
    Points: 7,821, Level: 59
    Level completed: 36%, Points required for next Level: 129

    Posts
    159
    Thanks
    1
    Thanked 7 Times in 7 Posts

    Re: New normed Goodness of Fit Measures ?

    Quote Originally Posted by hlsmith View Post
    Hmm, can you provide more details!


    Would a grouping of predictions (e.g., by decile) vs. observed values be of any benefit?
    No, it does not.
    Prediction is very difficult, especially about the future. (Niels Bohr)

  4. #4
    TS Contributor
    Points: 18,889, Level: 87
    Level completed: 8%, Points required for next Level: 461
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,062
    Thanks
    121
    Thanked 427 Times in 328 Posts

    Re: New normed Goodness of Fit Measures ?

    Quote Originally Posted by consuli View Post
    Hello everybody!

    I am currently working on a robust regression problem, where Model R^2 and Spearman correlation recommend different models.

    Are there new normed Goodness of Fit-/ Association-/ Correlation Measures away from the ones mentioned before (scale of target: interval)? With normed I am referring to at least a normed maximum fit (like 1 in R^2 and Correlation).

    Best case they will also work for robust problems.

    Thanks
    Consuli
    Can you explain more about why you want a normed measure? Why not do model selection via AIC, BIC, etc?
    Matt aka CB | twitter.com/matthewmatix

  5. #5
    Points: 7,821, Level: 59
    Level completed: 36%, Points required for next Level: 129

    Posts
    159
    Thanks
    1
    Thanked 7 Times in 7 Posts

    Re: New normed Goodness of Fit Measures ?

    Quote Originally Posted by CowboyBear View Post
    Can you explain more about why you want a normed measure? Why not do model selection via AIC, BIC, etc?
    Sure.

    I want to compare machine learning models (decision tree, neural network) with different kinds of regression models including interaction terms (OLS, GLM, Quantile, M-Estimator). Especially I want to know, to which proportion a simpler model (lets say OLS regression) has got the overall explanation of the best fitting modell. Thus, I need a normed measure.

    This all on robust data problems
    (target interval/ratio scale), error NOT following a classical theoretical distribution. As far I know, AIC and BIC make use of Maximum Likelihood, which is based always based on a theoretical distribution. Thus I am questioning wether they are reliable on robust data problems. Usually they are not recommended for robust problems.

    Maybe there are no new normed goodness of fit measures.

    Compare
    https://scholar.google.de/scholar?hl...ness+of+fit%22
    https://scholar.google.de/scholar?q=...e&as_sdt=0%2C5
    Prediction is very difficult, especially about the future. (Niels Bohr)

  6. #6
    TS Contributor
    Points: 18,889, Level: 87
    Level completed: 8%, Points required for next Level: 461
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,062
    Thanks
    121
    Thanked 427 Times in 328 Posts

    Re: New normed Goodness of Fit Measures ?

    Is this on simulated data or real data?
    Matt aka CB | twitter.com/matthewmatix

  7. #7
    Points: 3,006, Level: 33
    Level completed: 71%, Points required for next Level: 44

    Posts
    177
    Thanks
    1
    Thanked 29 Times in 29 Posts

    Re: New normed Goodness of Fit Measures ?

    I like to use Omega-naught-squared from Xu (2003). I think this might help a little bit. The measure of goodness of fit is a bit more robust than classical coefficients of determination.

    Here is the reference:

    Xu, R. (2003). Measuring explained variation in linear mixed effects models. Statistics in Medicine, 22(22), 3527-3541.

  8. #8
    Points: 7,821, Level: 59
    Level completed: 36%, Points required for next Level: 129

    Posts
    159
    Thanks
    1
    Thanked 7 Times in 7 Posts

    Re: New normed Goodness of Fit Measures ?

    Quote Originally Posted by CowboyBear View Post
    Is this on simulated data or real data?
    Both. I am using the robust datasets shipped in the R packages robustbase, robust and quantileregression. Further I am generating robust data by simulation.
    Prediction is very difficult, especially about the future. (Niels Bohr)

  9. #9
    Points: 7,821, Level: 59
    Level completed: 36%, Points required for next Level: 129

    Posts
    159
    Thanks
    1
    Thanked 7 Times in 7 Posts

    Re: New normed Goodness of Fit Measures ?

    Quote Originally Posted by the42up View Post
    I like to use Omega-naught-squared from Xu (2003). I think this might help a little bit. The measure of goodness of fit is a bit more robust than classical coefficients of determination.

    Here is the reference:

    Xu, R. (2003). Measuring explained variation in linear mixed effects models. Statistics in Medicine, 22(22), 3527-3541.
    Are you shure this omega-naught-squared is a general goodness-of-fit measure? In R it is located in the "Time-Domain Deconvolution of Seismometer Response" package (TDD). Compare https://cran.r-project.org/web/packages/TDD/TDD.pdf

    Looks strange to me.
    Prediction is very difficult, especially about the future. (Niels Bohr)

  10. #10
    TS Contributor
    Points: 18,889, Level: 87
    Level completed: 8%, Points required for next Level: 461
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,062
    Thanks
    121
    Thanked 427 Times in 328 Posts

    Re: New normed Goodness of Fit Measures ?


    Quote Originally Posted by consuli View Post
    Both. I am using the robust datasets shipped in the R packages robustbase, robust and quantileregression. Further I am generating robust data by simulation.
    For the simulated datasets you can assess how well the different methods do more directly - e.g., just estimate the bias of coefficients, calculate the mean squared deviation between the estimates of each parameter and its true value to estimate efficiency, etc. Not so sure about the real datasets..
    Matt aka CB | twitter.com/matthewmatix

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats