+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3
Results 31 to 43 of 43

Thread: Multilevel analysis -SAS

  1. #31
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS




    The Wald test for Random effects, the p value that tells you if a random effect is significant, is commonly seen as wrong. It is recommended that instead you add one random effect, look at the change in the model deviation, and do a chi square test (deviation model 1 -deviation of model 2) with a df of one using a chi square test.

    Does anyone know how to do this in SAS?

    This is one discussion of the approach

    Consider another example in which a model with a random effect for the slope (i.e, is the slope is allowed to vary) is compared to a model without the random effect for the slope (i.e., the variance
    of the slope is constrained). This example would appear to be testing a single parameter, but, infact, the two models differ by two parameters. The first model will include an estimate of the slope
    variance, τ 2 1, but also an estimate of the covariance between the slope and the intercept, τ10, by default. The covariance cannot be estimated when the slope is constrained to be non-varying,
    however. One would ordinarily expect that the difference between the two models would be compared to the chi-square distribution with df = 2, because two parameters differed between the
    models being compared. But because variance tests should use a one-tailed test and covariance tests are two-tailed tests, a more complicated significance criterion is needed. Snijders and Bosker
    (2012, p. 99) recommend using a "mixture distribution" (or "chi-bar distribution") by comparing the chi-square difference obtained from subtracting D0 – D1 to a combination of two critical values. For
    α = .05, the critical values are: one slope 2 χ mix = 5.14, two slopes 2 χ mix = 7.05, and three slopes 2 χ mix = 8.76.

    http://web.pdx.edu/~newsomj/mlrclass...gnificance.pdf

    The way I interpret this is if you were testing one random effect you would run it first with the fixed but not random effect for a slope (that is for the slope fixed). Then run it with the random and fixed effect for that predictor and determine how the deviance differed. Then, using this deviance, and a df of 2, you would run a chi square test. If the result was greater than 5.14 you would conclude the random effect was significant at the .05 level.

    Or are you comparing the empty model to the model with a random and fixed effect for that predictor specified and using those two models to get the difference in the deviation (everything else would be the same I assume).
    Last edited by noetsi; 09-01-2017 at 06:25 PM.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  2. #32
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS

    You use the empty model to predict the interclass correlation. Is this the correct SAS model to do that? I am not sure if you specify the intercept as random or not.

    proc mixed data= work.test4 covtest noclprint;
    class unitid_pri ;
    model weeklyearnings_clo = /solution;
    random intercept /subject= unitid_pri;
    run;

    While I am at it the link below has a macro that performs the LR deviance test with mixed p values (both strongly recommended especially with random effects). One thing that is unclear to me is if you have to run the

    %include '\\cdc\private\mixture method pvalue macro1.sas'; macro always or whether this unique to the data the author is using. He never mentions this macro at all.

    The link http://analytics.ncsu.edu/sesug/2009...ardenheier.pdf
    Last edited by noetsi; 09-05-2017 at 12:58 PM.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  3. #33
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS

    I am running the following code in SAS

    proc mixed data=work.test4 method=ml covtest empirical
    noclprint ;
    class unitid_pri female ;
    model dv=female/ ddfm=contain s ;
    random intercept /subject=unitid_pri type=ar(1) s ;
    parms / ols;
    ods output FitStatistics=fm1 SolutionF=SFfm1 ;

    Regardless of any fixed effect I test, there is only one variable in the model, I always get that the Hessian matrix is not positive definitive (the model converges, but this issue remains).

    I have about 77 groups and 6000 cases. The parms/ols; statement sets a starting value which is one way recommended to deal with a non-positive Hessian matrix (but which made no difference).
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  4. #34
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Multilevel analysis -SAS

    I have gotten that error before. Can you post a screenshot of the log so I can put it into context.

    So if I got it right this is for female status predicting DV, with either repeat measures or people clustered in groups. And the groups have random intercepts but fixed slopes based on gender.
    Stop cowardice, ban guns!

  5. #35
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Multilevel analysis -SAS

    Does every group have males or female? Is there any sparsity (low counts)?
    Stop cowardice, ban guns!

  6. #36
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS

    This is all the log says about this matter

    NOTE: Convergence criteria met but final Hessian is not positive definite.

    Its female status predicting the DV (its looking at groups, but not repeated measures). The intercept is random, the only fixed effect is female.

    According to some authors the results are invalid if you get this comment.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  7. #37
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS

    Quote Originally Posted by hlsmith View Post
    Does every group have males or female? Is there any sparsity (low counts)?

    Every cell had at least 1 female. There were a very few cells that had low counts (one had 3, but the next lowest was 25 of which 7 were women).
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  8. #38
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Multilevel analysis -SAS

    I would try a different covariance structure to see if you can resolve the warning. Also, keep an eye on the AIC in each of these model.


    (UN), may help, or


    (CS).






    Stop cowardice, ban guns!

  9. #39
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Multilevel analysis -SAS

    If that doesn't work, read the following, if you haven't already. I remember coming across this back in the day. In particular the example about variability across classrooms and using repeated instead of random option. Though I would first mess with the variance/covariance structure. Even though Jake said he would probably not mess with the convergence criteria, I would perhaps tweak it a little to let the model run a little longer.


    Lastly per the classroom example, you may need to also accept there isn't much variation explained by the group variable that isn't picked up in a simple model. Also what is your group variable? is it a geographic location, if so, locations may be getting separated but they are tangential and actually more similar than believed.


    Link:


    http://www.theanalysisfactor.com/wacky-hessian-matrix/
    Last edited by noetsi; 09-27-2017 at 11:15 AM.
    Stop cowardice, ban guns!

  10. The Following User Says Thank You to hlsmith For This Useful Post:

    noetsi (09-27-2017)

  11. #40
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS

    Quote Originally Posted by hlsmith View Post
    I would try a different covariance structure to see if you can resolve the warning. Also, keep an eye on the AIC in each of these model.


    (UN), may help, or

    (CS).






    My concern is that what I used is part of a macro needed to run the deviance test. I do not know how changing this effect this macro.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  12. #41
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Multilevel analysis -SAS

    Quote Originally Posted by hlsmith View Post
    If that doesn't work, read the following, if you haven't already. I remember coming across this back in the day. In particular the example about variability across classrooms and using repeated instead of random option. Though I would first mess with the variance/covariance structure. Even though Jake said he would probably not mess with the convergence criteria, I would perhaps tweak it a little to let the model run a little longer.


    Lastly per the classroom example, you may need to also accept there isn't much variation explained by the group variable that isn't picked up in a simple model. Also what is your group variable? is it a geographic location, if so, locations may be getting separated but they are tangential and actually more similar than believed.


    Link:


    http://www.theanalysisfactor.com/wacky-hessian-matrix/
    I accidently edited you comments rather than quoting them.... my apology. I think I put it back in its original form.


    My group is a geographical unit. The ICC was 3.7%, I don't know if that is a lot, little or whatever....

    My random effects do not show the problem with the Hessian matrix. Only the estimate of the fixed effects.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  13. #42
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Multilevel analysis -SAS

    Just quit using that Macro and go on with your life!
    Stop cowardice, ban guns!

  14. #43
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Multilevel analysis -SAS


    You are controlling for covariance in the region, though there is also covariance between tangential regions.
    Stop cowardice, ban guns!

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats