G Theory / Variance Components problems

[Uwyze]

I'm running SPSS Variance Components analyses using REML estimation. I have both real and fictitious data in which employees are rated by multiple raters. Subject ID and Rater ID are both treated as random.

On both real and fictitious data, the main effects seem to work ok. However, when I add the interaction between subject_ID and rater_ID to test for subject x rater effects, I get output where everything is .000 and a note says that the error component "is set to zero because it is redundant."

I followed up by running a univariate GLM which indicated the SS(error) and df for the interaction were both 0. I am assuming that may be related to the variance components errors, but I don't know how to work around the issue.

I've encountered this problem on a variety of different variables in both real and fictitious data. Any help would be greatly appreciated!

 

[Dason]

Does a rater ever rate the same employee more than once?

 

[Uwyze]

In that dataset they do not. Here is what happens in a different dataset where they do!

 

[Dason]

In this new dataset is there more than one assessor?

It seems to me that you're just having difficulty conceptualizing which quantities are going to be estimable. You need to have replicates for the random components otherwise you can't estimate a variance.

 

[Jake]

Is each subject rated by a different group of raters, or is each subject rated by the same group of raters? If it is the latter, do the labels for "ID_Assessor" in the dataset correctly reflect this? (I.e., the same rater should always have the same label for ID_Assessor).

 

[GretaGarbo]

You can make a cross table with raters and subjects. Do that cross table have at least two observations in each cell?

Does the title name refer to "generalizability theory"?

 

[Uwyze]

I think you might be right. I will play around with that idea in my simulated data. Thanks!

 

[Uwyze]

Yeah that seems to be correctly labeled - thanks!

 

[Uwyze]

You can make a cross table with raters and subjects. Do that cross table have at least two observations in each cell?

Does the title name refer to "generalizability theory"?

Thanks - this is a helpful suggestion. Yes, G Theory referred to Generalizability Theory.

 

[Uwyze]

Darn, I'm still missing something. I've created a dataset in which 20 participants are rated by 3 raters, twice each (Excel version attached). So we have a Participant (20) x Rater (3) x Time (2) design, which I think should put enough observations in each cell, correct?

I also tried to manipulate the sources of variance (e.g., in one column, all variance is between-persons; in another, raters vary but are consistent across time, etc.). I've also added random error to make sure there is at least a little bit of variance on each facet. However, when I run the variance components analyses...I get all zeros again.

What am I missing? Thanks so much!