1. ## Re: Multilevel analysis -SAS

The Wald test for Random effects, the p value that tells you if a random effect is significant, is commonly seen as wrong. It is recommended that instead you add one random effect, look at the change in the model deviation, and do a chi square test (deviation model 1 -deviation of model 2) with a df of one using a chi square test.

Does anyone know how to do this in SAS?

This is one discussion of the approach

Consider another example in which a model with a random effect for the slope (i.e, is the slope is allowed to vary) is compared to a model without the random effect for the slope (i.e., the variance
of the slope is constrained). This example would appear to be testing a single parameter, but, infact, the two models differ by two parameters. The first model will include an estimate of the slope
variance, τ 2 1, but also an estimate of the covariance between the slope and the intercept, τ10, by default. The covariance cannot be estimated when the slope is constrained to be non-varying,
however. One would ordinarily expect that the difference between the two models would be compared to the chi-square distribution with df = 2, because two parameters differed between the
models being compared. But because variance tests should use a one-tailed test and covariance tests are two-tailed tests, a more complicated significance criterion is needed. Snijders and Bosker
(2012, p. 99) recommend using a "mixture distribution" (or "chi-bar distribution") by comparing the chi-square difference obtained from subtracting D0 – D1 to a combination of two critical values. For
α = .05, the critical values are: one slope 2 χ mix = 5.14, two slopes 2 χ mix = 7.05, and three slopes 2 χ mix = 8.76.

http://web.pdx.edu/~newsomj/mlrclass...gnificance.pdf

The way I interpret this is if you were testing one random effect you would run it first with the fixed but not random effect for a slope (that is for the slope fixed). Then run it with the random and fixed effect for that predictor and determine how the deviance differed. Then, using this deviance, and a df of 2, you would run a chi square test. If the result was greater than 5.14 you would conclude the random effect was significant at the .05 level.

Or are you comparing the empty model to the model with a random and fixed effect for that predictor specified and using those two models to get the difference in the deviation (everything else would be the same I assume).

2. ## Re: Multilevel analysis -SAS

You use the empty model to predict the interclass correlation. Is this the correct SAS model to do that? I am not sure if you specify the intercept as random or not.

proc mixed data= work.test4 covtest noclprint;
class unitid_pri ;
model weeklyearnings_clo = /solution;
random intercept /subject= unitid_pri;
run;

While I am at it the link below has a macro that performs the LR deviance test with mixed p values (both strongly recommended especially with random effects). One thing that is unclear to me is if you have to run the

%include '\\cdc\private\mixture method pvalue macro1.sas'; macro always or whether this unique to the data the author is using. He never mentions this macro at all.

3. ## Re: Multilevel analysis -SAS

I am running the following code in SAS

proc mixed data=work.test4 method=ml covtest empirical
noclprint ;
class unitid_pri female ;
model dv=female/ ddfm=contain s ;
random intercept /subject=unitid_pri type=ar(1) s ;
parms / ols;
ods output FitStatistics=fm1 SolutionF=SFfm1 ;

Regardless of any fixed effect I test, there is only one variable in the model, I always get that the Hessian matrix is not positive definitive (the model converges, but this issue remains).

I have about 77 groups and 6000 cases. The parms/ols; statement sets a starting value which is one way recommended to deal with a non-positive Hessian matrix (but which made no difference).

4. ## Re: Multilevel analysis -SAS

I have gotten that error before. Can you post a screenshot of the log so I can put it into context.

So if I got it right this is for female status predicting DV, with either repeat measures or people clustered in groups. And the groups have random intercepts but fixed slopes based on gender.

5. ## Re: Multilevel analysis -SAS

Does every group have males or female? Is there any sparsity (low counts)?

6. ## Re: Multilevel analysis -SAS

NOTE: Convergence criteria met but final Hessian is not positive definite.

Its female status predicting the DV (its looking at groups, but not repeated measures). The intercept is random, the only fixed effect is female.

According to some authors the results are invalid if you get this comment.

7. ## Re: Multilevel analysis -SAS

Originally Posted by hlsmith
Does every group have males or female? Is there any sparsity (low counts)?

Every cell had at least 1 female. There were a very few cells that had low counts (one had 3, but the next lowest was 25 of which 7 were women).

8. ## Re: Multilevel analysis -SAS

I would try a different covariance structure to see if you can resolve the warning. Also, keep an eye on the AIC in each of these model.

(UN), may help, or

(CS).

9. ## Re: Multilevel analysis -SAS

If that doesn't work, read the following, if you haven't already. I remember coming across this back in the day. In particular the example about variability across classrooms and using repeated instead of random option. Though I would first mess with the variance/covariance structure. Even though Jake said he would probably not mess with the convergence criteria, I would perhaps tweak it a little to let the model run a little longer.

Lastly per the classroom example, you may need to also accept there isn't much variation explained by the group variable that isn't picked up in a simple model. Also what is your group variable? is it a geographic location, if so, locations may be getting separated but they are tangential and actually more similar than believed.

http://www.theanalysisfactor.com/wacky-hessian-matrix/

10. ## The Following User Says Thank You to hlsmith For This Useful Post:

noetsi (09-27-2017)

11. ## Re: Multilevel analysis -SAS

Originally Posted by hlsmith
I would try a different covariance structure to see if you can resolve the warning. Also, keep an eye on the AIC in each of these model.

(UN), may help, or

(CS).

My concern is that what I used is part of a macro needed to run the deviance test. I do not know how changing this effect this macro.

12. ## Re: Multilevel analysis -SAS

Originally Posted by hlsmith
If that doesn't work, read the following, if you haven't already. I remember coming across this back in the day. In particular the example about variability across classrooms and using repeated instead of random option. Though I would first mess with the variance/covariance structure. Even though Jake said he would probably not mess with the convergence criteria, I would perhaps tweak it a little to let the model run a little longer.

Lastly per the classroom example, you may need to also accept there isn't much variation explained by the group variable that isn't picked up in a simple model. Also what is your group variable? is it a geographic location, if so, locations may be getting separated but they are tangential and actually more similar than believed.

http://www.theanalysisfactor.com/wacky-hessian-matrix/
I accidently edited you comments rather than quoting them.... my apology. I think I put it back in its original form.

My group is a geographical unit. The ICC was 3.7%, I don't know if that is a lot, little or whatever....

My random effects do not show the problem with the Hessian matrix. Only the estimate of the fixed effects.

13. ## Re: Multilevel analysis -SAS

Just quit using that Macro and go on with your life!

14. ## Re: Multilevel analysis -SAS

You are controlling for covariance in the region, though there is also covariance between tangential regions.