Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance analyses

#1
I am running models looking at the fit of new self-report scale I have developed (CFA and measurement invariance by a number of different categories like gender, smoking status, etc.).

For CFA and MI analyses, I keep getting insanely good model fit (CFI = 1, RMSEA = 0, SRMR between .018 and .03). There are many degrees of freedom in the model, so I am not running into the problem of df = 1. When I run invariance analyses, CFI and RMSEA generally stay perfect, but SRMR changes (gets worse as expected, but not problematically). It would be great if we had a perfect model, but this seems suspect. Any thoughts on what could trigger this?

FYI, there is an invariance analysis (for race) where this is not the case. Fit for the configural model is RMSEA = .000, CFI = 1.014, SRMR = .19; Metric is RMSEA = .018, CFI = .997, SRMR = .42 and the scalar model is RMSEA = .038, CFI = .988, SRMR is .049.

Thanks for any thoughts you might have on this.

Meghan
 
#2
UPDATE: Model Fit is Too Perfect

I read somewhere online that modeling problems exist when the chi-square contribution is less than the degrees of freedom for any given step of a model (i.e., baseline fit for testing configural invariance or step comparing metric model to configural model, etc.) I have never run into this problem before and I do not really understand it. However, across the board, this seems to be the case for all of the models that have corresponding "perfect fit."

For example, when looking at the new measure by race, the fit for the configural model is RMSEA = .000, CFI = 1.000, SRMR = .19***; Metric is RMSEA = .027, CFI = .994, SRMR = .45 and the scalar model is RMSEA = .040, CFI = .982, SRMR is .050. It seems like the problem is with the con figural model (see red).

Invariance Testing


# Parameters; Chi-square; df; P-value

Config 30; 8.631; 10; 0.5675
Metric 26; 16.456; 14; 0.2863
Scalar 22; 25.093; 18; 0.1224

Do you have any sense at all what might account for this? Any thoughts you might have on this would be incredibly helpful.

Thanks,

Meghan
 

spunky

Can't make spagetti
#3
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

can you share your data and model? without actually tinkering with it, it's really hard to see what may be going on
 

Lazar

Phineas Packard
#4
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

Honestly I do not see the problem in this context. My guess is that your model is somewhat simple and the number of cases is relatively small. The model appears to be identified given that you have multiple degrees of freedom. Congratulations you have a well fitting model according to fit statistics (i.e. chi-square) and fit indices (i.e. RMSEA, CFI, etc.). That is to say there is nothing inherently wrong with those fit indices the only thing would be to consider whether the model you thought you were specifying is the model that you actually tested.
 

noetsi

No cake for spunky
#5
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

If your model is just identified, no df, you will get perfect fits. But from what you say you do have df.
 

Lazar

Phineas Packard
#6
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

If your model is just identified, no df, you will get perfect fits. But from what you say you do have df.
Note op has positive df in all three models. 10 14 and 18 to be exact.
 

CB

Super Moderator
#7
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

I wouldn't really use the term perfect fit. As you may have grasped, an RMSEA of zero and a CFI of one does not mean there is no discrepancy between the sample and model-implied covariance matrices. Rather RMSEA will be zero and CFI will be one whenever the chi-square statistic is equal to or less than the degrees of freedom.

So the fit statistics indicate very good fit, but they don't necessarily mean the fit is perfect. Which hopefully resolves your concern that the model shouldn't fit perfectly.
 

spunky

Can't make spagetti
#8
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

I still wanna see model and dataaaaaa!!! :D
 
#9
Info on data, example syntax and output for model

Unfortunately, I cannot share my data because of IRB issues. However, I can give you some info about the variables and share my syntax and MPLUS output. I have data from 493 participants. Scores for each of the 5 items range from 0-112 and are normally distributed (with the exception of Item 1 which has a skewness statistic of 1).

Here is an example of my syntax (this is the model depicted in the attachment) for a model looking at MI by race.

USEVARIABLES ARE
Item_1
Item_2
Item_3
Item_4
Item_5;

MISSING ARE All(999);

GROUPING IS race (0= non-caucasian 1 = caucasian);

ANALYSIS: ESTIMATOR=MLR;
MODEL = CONFIGURAL METRIC SCALAR;

MODEL:
Scale by
Item_1
Item_2
Item_3
Item_4
Item_5;

Hopefully this will be helpful. Thank you to everyone for giving me a hand!

Meghan
 
#10
Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

.... Rather RMSEA will be zero and CFI will be one whenever the chi-square statistic is equal to or less than the degrees of freedom....
Hi CowboyBear!
Thank you for the good explanation for "perfect" fit indices. It seems the right explanation for some models I've run. Do you know any books or journal articles that explain why exactly RMSEA will be zero and CFI will be one whenever the chi-square statistic is equal to or less than the degrees of freedom? That would be very helpful for my master thesis :)