Is my model a good fit?

#1
Hello everyone,

I am being tortured with the following problem...

I have a set of crude mortality rates for ages 0 to 60. Let's call them Observed Rates.
I use a smoothing technique (cubic spline regression) on the crude mortality rates and so smoothed mortality rates are produced for ages 0 to 60. Let's call them Estimated Rates.

I want to know whether the Estimated Rates are a good representation of the Observed Rates. I have calculated the R-square, however I am also looking to confirm this with a statistical test.

I am puzzled with which test I should pick, can you please help me out?
 

hlsmith

Not a robit
#2
Usually there isn't anything better than a visualization. Post a figure showing expected overlaid on observed. Statistical tests can be sample size dependent and misleading at times. Though they can provide support for decision made via visualizations.
 
#3
I am attaching below a graphical representation. The white bullets are the Observed Rates, while the red line corresponds to the Estimated Rates.
I guess one can say that graphically there seems to be a good fit, however I still need to perform a statistical test.

1537967170100.png
 

hlsmith

Not a robit
#4
EDITED: So you have rates across 10 year age groupings. N-values for groupings would be beneficial. Why do you want to fit a spline, which is difficult to interpret in comparison to reporting rates with confidence intervals?
 
Last edited:
#5
Exactly, 10 year age groupings. Fitting a spline is the given approach, I can't change this unfortunately. Could wilcoxon signed rank test work for my case?
 

ondansetron

TS Contributor
#6
Is there any variation in observed mortality within each age group or is there only a single value that applies to all observations in a bracket?
 

hlsmith

Not a robit
#8
Sorry for dragging my feet with questions, but is this just for creating a visualization? I wonder what a loess curve may generate. Did you tune the spline fitting or was it automated? I kind of wonder what it would look like if you just used the center value for the 10 year goupings (e.g. 5, 15, 25,...,) only also you don't have the full septuagenarian decade, which you could extend margin of fig to 80 and add half decade tick for better aesthetics.

Fit seems fine if it is just for visualization. Death is kind of like falls across time - represented by a pseudo-bathtube distribution.
 
#9
No problem at all with the questions. Nope I have not tried a loess fitting. Mainly because this is not required (cubic spline smoothing is requested) and also due to time restraints.

I am using the smooth.spline function in R and I am setting the number of knots with it so that one knot is placed in each age-band. Apart from the knot-setting thing the rest of it is automated. Whether the knot is placed on age 5 and 15 or age 7 and 17, I don't think influences much the resulted curve.

Anyhow... after reading and re-reading multiple sources on the web, I am inclined towards wilcoxon signed rank test for assessing the goodness of fit. Do you believe that's ok or do you have to suggest any other alternative?
 
#10
So you are proposing to subtracting the observed by the expected values and comparing the median different to a null value?

You may get into a little trouble with the following, given multiple r(similar) rate across the age groups, but this reminds me of the Hosmer-Lemeshow test for fit in logistic regression. Perhaps you can explore that.

The real question is why do you have to perform a standard test, Visualizations traditionally trump tests.
 
#11
Just thinking, do you only have aggregate level data? If not could you hold out some the fit the curve based on best fit for held out data?
 
#13
So you are proposing to subtracting the observed by the expected values and comparing the median different to a null value?
Yes, that's what I am thinking of, however I am not able to clearly tell if this test is "proper" for my case.

The real question is why do you have to perform a standard test, Visualizations traditionally trump tests.
I am asked to provide a proof of goodness-of-fit via statistical test, R-square and visual graphs will also provided.

If this is a mortality rate, how is there no variation within each age group? Trying to understand the data generation.
The rate is produced on a per age band basis. The database consists of age bands (0-9, 10-19...,70-79) and for each age-band the deaths and exposure are provided. Dividing the deaths by the exposure results in the mortality rate. So we end up with one mortality rate per age band. We are asked to fit a cubic spline so that it produces age-specific mortality rates (0,1,2,3,...,79). And then we are asked to test the goodness-of-fit with R-square and a statistics test.

p.s. thank you both a lot for taking the time to review my problem
 
Last edited:
#14
Is this an assignment, if so you can default to performing a test I suppose.

Another take on this may be conducting a survival curve, I would image they have goodness of fit tests.
 

ondansetron

TS Contributor
#16
It's a single value, no variation. For example, ages 0-9: rate 5% | ages 10-19: rate 3% | and so on...
If this is a mortality rate, how is there no variation within each age group? Trying to understand the data generation.
Yes, that's what I am thinking of, however I am not able to clearly tell if this test is "proper" for my case.



I am asked to provide a proof of goodness-of-fit via statistical test, R-square and visual graphs will also provided.



The rate is produced on a per age band basis. The database consists of age bands (0-9, 10-19...,70-79) and for each age-band the deaths and exposure are provided. Dividing the deaths by the exposure results in the mortality rate. So we end up with one mortality rate per age band. We are asked to fit a cubic spline so that it produces age-specific mortality rates (0,1,2,3,...,79). And then we are asked to test the goodness-of-fit with R-square and a statistics test.

p.s. thank you both a lot for taking the time to review my problem
So if there are artificial groupings and the rate is for an age group, how are there multiple observations per age group? Sorry if I missed something.
 
#17
The dataset is as follows (not exact figures):

Age Band ---- Mortality Rate
0-9 ----------------5%
10-19 ----------------3%
20-29 ----------------7%
30-39 ----------------7%
40-49 ----------------9%
50-59 ----------------12%
60-69 ----------------25%
70-79----------------35%

There are not multiple observations for each group. Just one, as shown above. Please let me know if anything is not clear.
 

ondansetron

TS Contributor
#18
I am attaching below a graphical representation. The white bullets are the Observed Rates, while the red line corresponds to the Estimated Rates.
I guess one can say that graphically there seems to be a good fit, however I still need to perform a statistical test.

View attachment 502
I read this wording and looked at the graph, which seemed to show multiple measurements per group, my fault.
 
#20
I would use one values within age ranges and explore the Hosmer-Lemeshow test that is used to examine logistic regression fit (predicted versus observed).