Multilevel analysis: Nesting variables to participants and comparing groups: what test to use?

#1
Hi all,
Hopefully, somebody can help me. I've been struggling for two weeks now on how to analyze the data for my thesis.
The design of the original study is as follows: 38 participants filled in multiple questionnaires and did a memory test in the evening. During the night, they were awoken nine times and asked to report everything that was going through their mind before they woke, and were asked a few other questions after this. After the final awakening in the morning, they did the memory test again.
Now, I did exploratory analyses on all data and decided that the aim of my study would be to explore/examine differences between anxious and non-anxious participants in information processing during the night (so the content of their dreams, etc.)
For most variables, like the questionnaires that were only taken once, this is relatively easy: I can perform a t-test on the dependent variables.
Since there are many dependent variables and just one categorical predictor I am interested in (anxious vs. non-anxious) I have given thought to performing a MANOVA, but I found online that this is discouraged when the dependent variables are not correlated. In my specific case, they aren’t, or at least not all of them. So I guess I would have to perform many t-tests. (tips on this are welcome as well, regarding type I error increasing with doing multiple tests and correcting for this).
However, my main problem is how to analyze the data from the awakenings. For some participants, there are a few awakenings missing because they slept through the phone call, or because an awakening was skipped, or in some cases because the experimenter accidentally fell asleep. This results in different numbers of awakenings per participant, which rules out a repeated measures ANOVA.
But comparing all awakenings from the two different anxiety groups is also not the way to go, since then the between-subject variance is lost.
I've tried using a Mixed Model analysis, but when I select both participants and awakenings as random effects, and anxiety and one of the dependent variables of interest as fixed effects, I have trouble interpreting the results and it seems as if the different awakenings are compared, which is not what I want.
Instead, I'm looking for a way to "nest" or "weigh" the awakenings hierarchically to each participant. I want to compare the different variables that were measured at each awakening between the groups, without discarding between-subject differences. What statistical model can I use for this?
I'm sorry for the long story. Hopefully there's somebody who can help me. I can also send the R-scripts of what I've been doing so far so we can discuss together how to interpret the results and what I can change or do. I've been using the lme & lmer functions, but the output confuses me.
 
Last edited:

noetsi

Fortran must die
#2
I know little about MANOVA but if you have many predictors and one dependent variable why not use either multiple regression or ANOVA. MANOVA is used when you want to predict more than one dependent variable. There are ways to deal with missing data if its not too extensive.

Multilevel models are usually nested inside something. I am not sure what is nested here.
 
#3
I know little about MANOVA but if you have many predictors and one dependent variable why not use either multiple regression or ANOVA. MANOVA is used when you want to predict more than one dependent variable. There are ways to deal with missing data if its not too extensive.

Multilevel models are usually nested inside something. I am not sure what is nested here.
Oh, I’m so sorry. I confused the definitions of the independent and dependent variables. After three years of university and four courses on statistics, I still haven’t mastered the lexicon. I edited my post, hopefully it is more clear what I mean now.
There is just one independent categorical predictor (anxiety), and many dependent variables (scores on different questionnaires, pre- and post-sleep memory scores, the number of reported threats in dreams, the number of dreams related to personal events, etc. etc.). Some of these dependent variables are correlated, and some are unrelated. Should I perform a MANOVA on the dependent variables that correlate, and additional t-tests for all unrelated dependent variables?
 

noetsi

Fortran must die
#4
Don't feel bad I have spent a decade after many degrees touching on it and still don't understand it. :p
I don't know enough about MANOVA to give advice. Except that it is not multilevel models as I have seen that later term used (where some phenomenon is nested inside another, say students in classes). You might want to move this to the psychology thread, it looks like psychology to me. Generally speaking lots of independent t tests are not a great idea.
 

Karabiner

TS Contributor
#5
Now, I did exploratory analyses on all data and decided that the aim of my study would be to explore/examine differences between anxious and non-anxious participants in information processing during the night (so the content of their dreams, etc.)
If you decided so on basis interestingly-looking descriptive statistics,
this could hugely devaluate all further findings, though.
Since there are many dependent variables and just one categorical predictor I am interested in (anxious vs. non-anxious)
In most studies, we do not deal with clearly separated groups of
anxious and non-anxious participants, but researches commit the
fault of dividing subjects into 2 adjacent classes, based on
sample medians, or based on some other arbitraty cuptoff. This
practice has logical problems (why is someone at the median considered
as categorically different from a near neighbur, but the "same" as
someone with an extreme value), wastes statistical information,
and might lead to misleading and/or irreproducable results.
https://statisticalanalysisconsulting.com/the-perils-of-categorizing-continuous-variables/
http://www.psychology.sunysb.edu/attachment/measures/content/maccallum_on_dichotomizing.pdf

If you did somethng like this, but still have the original anxiety scores,
then you should use those instead of arbitrary groups. Technically,
this is not diffcult.

I have given thought to performing a MANOVA, but I found online that this is discouraged when the dependent variables are not correlated.
For MANOVA, dependent variables shoud jointly represent the
same theoretical construct, and should be moderately correlated
(if highly correlated, then they're redundant). The first aspect is
important.
I've tried using a Mixed Model analysis, but when I select both participants and awakenings as random effects, and anxiety and one of the dependent variables of interest as fixed effects,
The multileve approach makes sense. Why is the dependent variable a fixed effect?

With kind regards

Karabiner
 
#6
Don't feel bad I have spent a decade after many degrees touching on it and still don't understand it. :p
I don't know enough about MANOVA to give advice. Except that it is not multilevel models as I have seen that later term used (where some phenomenon is nested inside another, say students in classes). You might want to move this to the psychology thread, it looks like psychology to me. Generally speaking lots of independent t tests are not a great idea.
Thanks for the tip! I don't know how to move my post but I rewrote it partly and added it to the Psychology thread.
 
#7
If you decided so on basis interestingly-looking descriptive statistics,
this could hugely devaluate all further findings, though.

In most studies, we do not deal with clearly separated groups of
anxious and non-anxious participants, but researches commit the
fault of dividing subjects into 2 adjacent classes, based on
sample medians, or based on some other arbitraty cuptoff. This
practice has logical problems (why is someone at the median considered
as categorically different from a near neighbur, but the "same" as
someone with an extreme value), wastes statistical information,
and might lead to misleading and/or irreproducable results.
https://statisticalanalysisconsulting.com/the-perils-of-categorizing-continuous-variables/
http://www.psychology.sunysb.edu/attachment/measures/content/maccallum_on_dichotomizing.pdf

If you did somethng like this, but still have the original anxiety scores,
then you should use those instead of arbitrary groups. Technically,
this is not diffcult.


For MANOVA, dependent variables shoud jointly represent the
same theoretical construct, and should be moderately correlated
(if highly correlated, then they're redundant). The first aspect is
important.

The multileve approach makes sense. Why is the dependent variable a fixed effect?

With kind regards

Karabiner
Thanks a lot for your answer.
The reason I decided to look at anxiety as a group, was because when using a cut-off score based on the literature I could find, this was the only way to get two groups that were roughly of the same size (n=16 & n=22). When using statistical rules (like >2SD's from the mean, or cut-off scores for other questionnaires) this resulted in either very small gropus or unequal group sizes.
I indeed still have the original anxiety scores which I could use, but I'm afraid this will further complicate the analysis as I will no longer have a categorical predictor. Can a multilevel approach still be used with a continuous predictor? And would I then have to use a multiple regression analysis instead of a MANVOA for the other, 'single' dependent variables?
I thought the dependent variable had to be added as a fixed effect in order for the analysis to work... But since I've never worked with these kinds of analyses before, I just don't really know what I'm doing. An example of a code I've used in R (linear mixed model):
lme(dependentvariable1~anxietygroups*dependentvariable2,random=~1|anxietygroups/subject)
 

Karabiner

TS Contributor
#8
I indeed still have the original anxiety scores which I could use, but I'm afraid this will further complicate the analysis as I will no longer have a categorical predictor.
Arbitrary groups are a well-known mess, and an interval scaled predictor is no
more complicated than a binary variable, AFAICS.
Can a multilevel approach still be used with a continuous predictor?
You can call that a multilevel regression.
I thought the dependent variable had to be added as a fixed effect in order for the analysis to work...
But it is the dependent variable. The variable you want to
be explained. It is not an explaining variable.

With kind regards

Karabiner
 
#9
Arbitrary groups are a well-known mess, and an interval scaled predictor is no
more complicated than a binary variable, AFAICS.

You can call that a multilevel regression.

But it is the dependent variable. The variable you want to
be explained. It is not an explaining variable.

With kind regards

Karabiner
Thank you! I will try to familiarize myself with a multilevel regression.
 
#11
Or maybe it is even more easy than that. I forgot that "anxiety" does not vary across nights.
Do you mean I wouldn't have to use a multilevel approach?
When using the anxiety score, which is available at the participant-level, and using this as a predictor for variables at the awakening-level, I think I need a two-level model of awakenings clustered in participants? Or will this result in a comparison of the awakenings? Because that is not the goal.
 

Karabiner

TS Contributor
#12
You need a multilevel approach since your have mutliple observations for each participant, and repeated-measures-ANOVA is not useful because of missing values. Alternatively, you could consider aggregating the intra-individual measurements across nights, if that makes sense, and perform a simple mulitple regression with the aggregated values as DV. Or you impute the missing values and perfom repeatad-measures analysis of variance. Or you do a multilevel model. It will explain measurements at the awakenings level, using anxiety as predictor.

With kind regards

Karabiner