+ Reply to Thread
Results 1 to 10 of 10

Thread: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance analyses

  1. #1
    Points: 28, Level: 1
    Level completed: 56%, Points required for next Level: 22

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance analyses




    I am running models looking at the fit of new self-report scale I have developed (CFA and measurement invariance by a number of different categories like gender, smoking status, etc.).

    For CFA and MI analyses, I keep getting insanely good model fit (CFI = 1, RMSEA = 0, SRMR between .018 and .03). There are many degrees of freedom in the model, so I am not running into the problem of df = 1. When I run invariance analyses, CFI and RMSEA generally stay perfect, but SRMR changes (gets worse as expected, but not problematically). It would be great if we had a perfect model, but this seems suspect. Any thoughts on what could trigger this?

    FYI, there is an invariance analysis (for race) where this is not the case. Fit for the configural model is RMSEA = .000, CFI = 1.014, SRMR = .19; Metric is RMSEA = .018, CFI = .997, SRMR = .42 and the scalar model is RMSEA = .038, CFI = .988, SRMR is .049.

    Thanks for any thoughts you might have on this.

    Meghan

  2. #2
    Points: 28, Level: 1
    Level completed: 56%, Points required for next Level: 22

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    UPDATE: Model Fit is Too Perfect

    I read somewhere online that modeling problems exist when the chi-square contribution is less than the degrees of freedom for any given step of a model (i.e., baseline fit for testing configural invariance or step comparing metric model to configural model, etc.) I have never run into this problem before and I do not really understand it. However, across the board, this seems to be the case for all of the models that have corresponding "perfect fit."

    For example, when looking at the new measure by race, the fit for the configural model is RMSEA = .000, CFI = 1.000, SRMR = .19***; Metric is RMSEA = .027, CFI = .994, SRMR = .45 and the scalar model is RMSEA = .040, CFI = .982, SRMR is .050. It seems like the problem is with the con figural model (see red).

    Invariance Testing


    # Parameters; Chi-square; df; P-value

    Config 30; 8.631; 10; 0.5675
    Metric 26; 16.456; 14; 0.2863
    Scalar 22; 25.093; 18; 0.1224

    Do you have any sense at all what might account for this? Any thoughts you might have on this would be incredibly helpful.

    Thanks,

    Meghan

  3. #3
    TS Contributor
    Points: 22,355, Level: 93
    Level completed: 1%, Points required for next Level: 995
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

    can you share your data and model? without actually tinkering with it, it's really hard to see what may be going on
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  4. #4
    Phineas Packard
    Points: 16,013, Level: 81
    Level completed: 33%, Points required for next Level: 337
    Lazar's Avatar
    Location
    Sydney
    Posts
    1,159
    Thanks
    198
    Thanked 336 Times in 299 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

    Honestly I do not see the problem in this context. My guess is that your model is somewhat simple and the number of cases is relatively small. The model appears to be identified given that you have multiple degrees of freedom. Congratulations you have a well fitting model according to fit statistics (i.e. chi-square) and fit indices (i.e. RMSEA, CFI, etc.). That is to say there is nothing inherently wrong with those fit indices the only thing would be to consider whether the model you thought you were specifying is the model that you actually tested.
    "I have done things to data. Dirty things. Things I am not proud of."

  5. #5
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

    If your model is just identified, no df, you will get perfect fits. But from what you say you do have df.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  6. #6
    Phineas Packard
    Points: 16,013, Level: 81
    Level completed: 33%, Points required for next Level: 337
    Lazar's Avatar
    Location
    Sydney
    Posts
    1,159
    Thanks
    198
    Thanked 336 Times in 299 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

    Quote Originally Posted by noetsi View Post
    If your model is just identified, no df, you will get perfect fits. But from what you say you do have df.
    Note op has positive df in all three models. 10 14 and 18 to be exact.
    "I have done things to data. Dirty things. Things I am not proud of."

  7. #7
    TS Contributor
    Points: 18,889, Level: 87
    Level completed: 8%, Points required for next Level: 461
    CowboyBear's Avatar
    Location
    New Zealand
    Posts
    2,062
    Thanks
    121
    Thanked 427 Times in 328 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

    I wouldn't really use the term perfect fit. As you may have grasped, an RMSEA of zero and a CFI of one does not mean there is no discrepancy between the sample and model-implied covariance matrices. Rather RMSEA will be zero and CFI will be one whenever the chi-square statistic is equal to or less than the degrees of freedom.

    So the fit statistics indicate very good fit, but they don't necessarily mean the fit is perfect. Which hopefully resolves your concern that the model shouldn't fit perfectly.

  8. The Following User Says Thank You to CowboyBear For This Useful Post:

    Madita (06-22-2016)

  9. #8
    TS Contributor
    Points: 22,355, Level: 93
    Level completed: 1%, Points required for next Level: 995
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal

    I still wanna see model and dataaaaaa!!!
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  10. #9
    Points: 28, Level: 1
    Level completed: 56%, Points required for next Level: 22

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Info on data, example syntax and output for model

    Unfortunately, I cannot share my data because of IRB issues. However, I can give you some info about the variables and share my syntax and MPLUS output. I have data from 493 participants. Scores for each of the 5 items range from 0-112 and are normally distributed (with the exception of Item 1 which has a skewness statistic of 1).

    Here is an example of my syntax (this is the model depicted in the attachment) for a model looking at MI by race.

    USEVARIABLES ARE
    Item_1
    Item_2
    Item_3
    Item_4
    Item_5;

    MISSING ARE All(999);

    GROUPING IS race (0= non-caucasian 1 = caucasian);

    ANALYSIS: ESTIMATOR=MLR;
    MODEL = CONFIGURAL METRIC SCALAR;

    MODEL:
    Scale by
    Item_1
    Item_2
    Item_3
    Item_4
    Item_5;

    Hopefully this will be helpful. Thank you to everyone for giving me a hand!

    Meghan
    Attached Files

  11. #10
    Points: 4, Level: 1
    Level completed: 7%, Points required for next Level: 46

    Posts
    1
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Model Fit is Too Perfect: RMSEA & CFI are perfect; SRMR varies in invariance anal


    Quote Originally Posted by CowboyBear View Post
    .... Rather RMSEA will be zero and CFI will be one whenever the chi-square statistic is equal to or less than the degrees of freedom....
    Hi CowboyBear!
    Thank you for the good explanation for "perfect" fit indices. It seems the right explanation for some models I've run. Do you know any books or journal articles that explain why exactly RMSEA will be zero and CFI will be one whenever the chi-square statistic is equal to or less than the degrees of freedom? That would be very helpful for my master thesis

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats