# Thread: Correlational design using nominal/ordinal data

1. ## Re: Correlational design using nominal/ordinal data

Originally Posted by CowboyBear
If you have not yet come across any of the number of articles in psychology pointing out why p values do not tell us much about meaningfulness, I'd suggest starting with The Earth is Round (p < .05) by the brilliant Jacob Cohen.
Thanks for posting that. I found it to be a very enjoyable read and now I have a good paper to cite. I got a good laugh out of "I resisted the temptation to call it statistical hypothesis inference testing"

2. ## Re: Correlational design using nominal/ordinal data

When you reject the null hypothesis, you are accepting the experimental hypothesis,
Not exactely. Experimental hypothesis and statistical null
hypotheses are from different domains. For example,
experimental hypotheses usually (if they are not identical
with the null hypothesis) are no exact point hypotheses.
namely that there IS a significant difference between variables or conditions
Experimental hypotheses may state that there is a non-zero difference in the
populations where the data are sampled from. Whether this is significant in the
non-statistical sense (importance or relevance) cannot be answered by the
statistical significance test. The statistical significance test only provides
information about how probable sample data were if one assumed a true
null hypothesis (which ususally is: in the population the effect is EXACTELY
0.00). To reject the null, you either need a strong effect (if you're dealing with
small samples) or simply a large sample.
A simple ranking will not do the job, because it tells you nothing about the statistical significance of differences between conditions in the data.
Your research question was not "is there a significance of differences between
conditions in the data". I was referring to your original question. And ranking
would be just the start, testing would be the next step.
Ranking just tells me that one mean (or median) is greater than another, not about probabilities from which effects can be inferred.
What you mean by probabilities from which effects can be inferred, I am not sure.
One could test if rank differences within your sample permit inferences about rank differences in the population.

Kind regards

K.

3. ## Re: Correlational design using nominal/ordinal data

Originally Posted by CowboyBear
No, this is false (or at least: when some psychologists interpret p values this way, they are seriously mistaken).
I am not one of those psychologists.

Originally Posted by CowboyBear
I'm in psych myself, btw. A p value simply indicates that probability of observing a statistic as extreme as that observed given that the null hypothesis is exactly true. It says almost nothing about the meaningfulness of the finding. A correlation of 0.01 between two variables might be statistically significant if you have a big enough sample, but it may be so small as to be practically meaningless.
People seem to be more eager to misunderstand me and quibble over semantics than help me with my problem. I said that probability ratings are *a* fundamental criterion, not *the* exclusive criterion by which the meaningfuless of differences in data is decided. And yes people who rely purely on p numbers will come unstuck if they don't do their power calculations, and yes sample size and effect size are crucial but I thought this would go without saying in this community. I've spent two years as a Research Associate, so I do know a little bit about conducting research, however my grasp of non-parametric statistics is poor, my supervisor is on maternity leave, and I need help.

Nice to meet a fellow psych, perhaps you know how to fix my analysis. Excuse my tone but I'm in a crappy mood, as getting answers to a couple of simple questions is proving excruciating. I'm running behind schedule for ethics committee approval, and despite spending a fortune on textbooks and reading everything in existence on the treatment of categorical data I'm getting nowhere.

4. ## Re: Correlational design using nominal/ordinal data

Except it wasn't an answer to my question. It was a diversion into a straw man argument against people who don't understand effect sizes and sample sizes and rely exclusively on p-values.

5. ## Re: Correlational design using nominal/ordinal data

Originally Posted by Zenid
People seem to be more eager to misunderstand me and quibble over semantics than help me with my problem.
ok... couple of things. the problem is that these are not just "semantics". after reading the previous exchanges you've had with other contributors i also realize that if what you previously wrote reflects your understanding of null hypothesis testing, then it is, in fact, wrong. and i'm the third person here telling you this but that's beyond the point in terms of your question.... in which case... well... i'm stuck exactly where Karabiner is. every time i see a new post with what seems would be a solution you're proposing one of two things changes: either you change the research question or the analysis that you're putting forward implies a different research question.

first things first:

- on the title you say this pertains some sort of correlational design which immediatley made me think that either some sort of log-linear analysis would be in place.

- then in the following post you say that you want to compare means when you claim "This makes it easier to see what I need to do: I need to compare rows and test the hypothesis that "The mean of the happy row is significantly greater than the mean of all other rows" which i interpreted as "oh well, then maybe the real point here is just on comparing people across groups"

- then there's the post where you say "For the forced choice, it looks like a regular Chi-Square after all" which made me think "oh, so all this person wants to achieve is see if there is some sort of relaitonship between the pictures being shown and the words being described in the forced-choice section".

and then you go back to say this is a correlational design with categorical data.

all in all i'm exactly where Karabiner and CowboyBear are. i have absolutely no idea of what you're trying to do and when we dont understand what you're trying to do, we cant really help much.

oh, and just as a minor side-bar... when you say that a potential solution is to grab each column and do a separate chi-square analysis on it (but i think you didnt like it because it's not elegant) you need to keep in mind you'll increase your experimentwise error rate from running the same analysis over and over on the same data set so it will no longer be .05, you'll need to adjust for it depending on how many chi-square tests you plan on running...

6. ## The Following User Says Thank You to spunky For This Useful Post:

Zenid (02-01-2012)

7. ## Re: Correlational design using nominal/ordinal data

Originally Posted by spunky
i'm stuck exactly where Karabiner is. every time i see a new post with what seems would be a solution you're proposing one of two things changes: either you change the research question or the analysis that you're putting forward implies a different research question.
What is hard to understand? One person already said "oh you explained the problem real well but you'd better go away and show that you've done some work on the problem cos we're not going to do it all for you".

I've gone away and looked at the problem every which way and looked at different ways of treating and conceptualising it in the hope of earning some kind of guidance, but people still seem confused.

For "Design 1" I'm looking for CORRELATIONS between the independant variable (facial expression) and the dependant variable (choice of adjective selected by participants). Period. I went to the trouble of drawing up a grid sample data that makes the design obvious (Trinker said as much "You've done a pretty good job of giving us info but {go away and do some more work}").

Originally Posted by spunky
- on the title you say this pertains some sort of correlational design which immediatley made me think that either some sort of log-linear analysis would be in place.
A loglinear analysis is for categorical data of MORE THAN TWO variables. I am looking (in this example) at TWO variables broken into six categories.

Originally Posted by spunky
- then in the following post you say that you want to compare means when you claim "This makes it easier to see what I need to do: I need to compare rows and test the hypothesis that "The mean of the happy row is significantly greater than the mean of all other rows" which i interpreted as "oh well, then maybe the real point here is just on comparing people across groups"
Yes. That was the approach I was taking to address the analysis "Design 2", in which I'm not dealing with forced choice (where one person makes a choice between response adjectives), but a scenario where EVERY adjective is scored according to the degree to which it is perceived to concur with the facial expression on the stimuls photo (see table 3).

It's a different statistical treatment because it's no longer a frequency grid. A related design simply comparing row means will do the job, but the problem is I then have to compare means for responses like so:

"Happy" photos: compare responses to HAPPY" adjective to "surpised" scores, then to "sad" scores, then to "surprised" scores, "shy" scores, "tired" scores, "afraid" scores.

"Suprised" photos: compare responses to "SURPRISED" adjective to "surpised" scores, then to "sad" scores, then to "surprised" scores, "shy" scores, "tired" scores, "afraid" scores.

etc. etc., do you see? But the problem is of course I end up doing tests for every combination. Which is not elegant (I have to divide p ratings by the number of tests I do too, in the manner you described).

Originally Posted by spunky
- then there's the post where you say "For the forced choice, it looks like a regular Chi-Square after all" which made me think "oh, so all this person wants to achieve is see if there is some sort of relaitonship between the pictures being shown and the words being described in the forced-choice section".
YES! That's for the FIRST Design composed of FORCED-CHOICE responses.

Originally Posted by spunky
and then you go back to say this is a correlational design with categorical data.
Yes, because I'm referring to Design 1 again.

Sorry, I can see why it's getting confusing. I'm trying to address both designs at the same time and I've not made that clear. My apologies.

Originally Posted by spunky
oh, and just as a minor side-bar... when you say that a potential solution is to grab each column and do a separate chi-square analysis on it (but i think you didnt like it because it's not elegant) you need to keep in mind you'll increase your experimentwise error rate from running the same analysis over and over on the same data set so it will no longer be .05, you'll need to adjust for it depending on how many chi-square tests you plan on running...
YES again!!! I thought of that, but it's not very elegant at all. Plus I can't figure out how to feed it into SPSS

I was hoping somebody might know of some clever trick where I can tell SPSS to treat the columns as related, as they deal with the same SAME participants making choices between different responses (rows), NOT a regular chi-square where each participant fits only into one box of all rows and columns.

Sorry again if my train-of-thought posts were confusing. Don't be offended if I'm abrupt, I can't help it. I'm absolutely frazzled.

Best wishes, and thanks to everyone for chipping in!

8. ## Re: Correlational design using nominal/ordinal data

Originally Posted by CowboyBear
If you have not yet come across any of the number of articles in psychology pointing out why p values do not tell us much about meaningfulness, I'd suggest starting with The Earth is Round (p < .05) by the brilliant Jacob Cohen.
The journal articles and textbooks I'm reading explain just fine why this *alone* isn't good enough, but I'll read the article anyway as it looks like a goodie and this is important. Thanks.

9. ## Re: Correlational design using nominal/ordinal data

Originally Posted by Zenid
What is hard to understand?
Perhaps I'm being ever so dull, but the reasons I find this all hard to understand are the same reasons spunky mentions - the changing research questions.

To show what I mean... Your initial objectives were to:

Originally Posted by Zenid
1) assess which pictures are most clearly matched with a single descriptor word.
2) identify which set of pictures, that is, which human face, most successfully displays the full range of expressions.
There is nothing particularly wrong with these questions, but they are not questions about correlations. Then you say:

Originally Posted by Zenid
When I say I need to "assess which pictures are most clearly matched with a single descriptor word" I mean that I must show which descriptor or descriptors are selected at a rate significantly above chance expectation for a given stimulus picture.
But these are two completely different questions. Here it seems like you're coercing a perfectly interesting original question into a revised form where you can specify and test a null hypothesis... but losing the original intent of the question along the way. I would also suggest tentatively that the second question in the quote above is not a particularly meaningful one - a null hypothesis that participants match descriptors to pictures completely randomly is not at all plausible, so I'm not sure there's much point in testing it.

To be honest, I think what you really need to do is make a firm decision about you are actually trying to find out here. Your original questions looked workable, although they don't necessarily lead neatly to a particular choice of mainstream statistical test. That's ok though! Better to choose an analysis that actually answers your questions (even if it has to be a bit creative and different), then to coerce your research questions into matching a particular statistical test.

Originally Posted by Zenid
Originally Posted by CowboyBear
If you have not yet come across any of the number of articles in psychology pointing out why p values do not tell us much about meaningfulness, I'd suggest starting with The Earth is Round (p < .05) by the brilliant Jacob Cohen.
The journal articles and textbooks I'm reading explain just fine why this *alone* isn't good enough, but I'll read the article anyway as it looks like a goodie and this is important. Thanks.
It is a goodie, even just for the sake of it being a lot more entertaining than most journal articles! As/if you read more about this issue, though, you'll find that the debate is more than just about whether p values need to be supplemented with other information: It's also about whether they give any useful information at all. There are a number of psychologists (and scientists in other fields) calling for replacing p values altogether. E.g.: A practical solution to the pervasive problems of p values.

10. ## Re: Correlational design using nominal/ordinal data

Just as an aside (so you may well skip reading this):
I went to the trouble of drawing up a grid sample data that makes the design obvious (Trinker said as much "You've done a pretty good job of giving us info but {go away and do some more work}").
Seems as if you did the job too well. The way in which you displayed
on the other hand, that crosstabulation approach misleads the search
for a data analytical strategy. Instead of looking for an analysis which
to properly analyse crosstables. Interesting and important experience.

Kind regards

11. ## Re: Correlational design using nominal/ordinal data

Originally Posted by CowboyBear
To be honest, I think what you really need to do is make a firm decision about you are actually trying to find out here. Your original questions looked workable, although they don't necessarily lead neatly to a particular choice of mainstream statistical test. That's ok though! Better to choose an analysis that actually answers your questions (even if it has to be a bit creative and different), then to coerce your research questions into matching a particular statistical test.
This is exactly the problem. I'm TRYING to come up with a treatment that answers the original question, - this, remember, is in response to the perfectly fair request that I go away and try and figure it out for myself before coming back to you for feedback on my treatment. (I'm talking about "Design 1" and my chi-square approach for now).

But now I've come back with a vaguely plausible-looking but flawed treatment for at least the first question, I'm having problems understanding exactly why you object to it.

I thought it was a perfect example of a correlational design in the sense that I'm looking for a "correlation matrix" that tells me the strength of association between each column item and each row item (much like the multi-trait multi-method approach used in some of the studies I'm looking at). Let's put aside for the moment the complexities of inference of 'meaningfulness' from P values (figuring what sample size I need given the effect size I'm looking for and the power I want from my test is the next stage once I HAVE a viable test).

I thought the extent to which individual cell values depart from chance expectation are a classic example of a measure of the degree to which row and column categories "correlate" (in sofar as the term is technically acceptable in dealing with categorical data). If the "happy photo"/"happy word" cell indicates a vast majority are correctly choosing this word, then related cell residuals are at least a broad measure of the "strength" of the match, compared to other cell correspondences. I need these values (or at least think I do) to give me a results grid that both shows me the relative strengths of these cross-comparisons, and ALSO gives me a p-value that (along with sample/power/effect considerations) gives me an idea of which can be considered significant in the context of the data.

Both you and Karibiner voice the same concerns, saying that there is some kind of disconnect between my research question and my (attempted) chi-square treatment, and even the basic principle of my treatment and the conception of 'correlation' here.

This presumably isn't just antipathy towards the use of p-values in psychology, - you both know (and so does Jacob) that we're stuck with them, and can only emphasise the importance of effect size and get authors to report them more as part of their research to try and standardise things.

If it's more than just that, then I'm just not grasping you point - where is the disconnect? What am I missing that's getting me so confused about your objection?

More importantly, what do you see as the RIGHT way of addressing my (first question)?

12. ## Re: Correlational design using nominal/ordinal data

Originally Posted by Karabiner
Just as an aside (so you may well skip reading this):
Seems as if you did the job too well. The way in which you displayed
on the other hand, that crosstabulation approach misleads the search
for a data analytical strategy.
This is where I'm confused. I thought a basic crosstab strategy was tailor-made for this kind of design (see my response to coyboy bear). I thought that chi-square was in the right ball-park at least for treating my hypothetical data. This impression was largely inheritted from my professor who came out with "oh that looks like a chi-square" by way of getting me started, then sent me away to figure it the rest for myself.

I've now read and re-read chapters on the treatment of categorical data in "Research Methods and Statistics in Psychology"and "Research Methods in Psychology." I've bought two more books "Discovering statistics using SPSS" and "SPSS for Psychologists" and they all lead me down to a chi-square dead-end where my related design (100 participants judging ALL face-pictures across the columns) gets interpreted as a unrelated design that treats my grid as 600 participants divided equally across columns.

Where is this disconnect is confusing both you and cowboybear? Are you absolutely sure that it's not simply due to the area of research that I'm involved ("soft psychology") and the 'cultural' differences relating to the way stats are applied in a given area?

Either there's something I'M missing about the way I'm trying to treat my data or there's something YOU'RE missing about the way psychology "does business".

Originally Posted by Karabiner
Instead of looking for an analysis which
to properly analyse crosstables. Interesting and important experience.
I have searched high and low and this is all I have come up with. I blew \$100 on textbooks to "start from scratch" in going back to first principles and figuring out what to do with this.

I'm stuck. Period

13. ## Re: Correlational design using nominal/ordinal data

Hi again. I'm still pondering this and trying to construct an actually useful suggestion for an analytic approach, but perhaps for the moment I can explain why I don't see your questions as being about correlations.

There's a lot of ways to conceptualise correlations, but one basic way to look at them is that a correlation is a measure of the relationship between variation in X and variation in Y. If X increases, does Y tend to increase too, and vice versa?

However, if we look at your first question:

1) assess which pictures are most clearly matched with a single descriptor word.
...this isn't directly a question about relationships between a pair of variables (or pairs of variables). It's certainly not like you're asking "if picture increases, does descriptor tend to increase too?" It's really more of a question about identifying a set of stimulus objects that have a particular property.

To make this a little clearer, I might rephrase your question a little. I'm going to assume that each pictured face is actually intended to express one of your listed emotions (stop me if this is wrong!) So:

"For which pictures is the proportion of responses selecting the hypothesised emotional descriptor the highest?"

Originally Posted by Zenid
I thought it was a perfect example of a correlational design in the sense that I'm looking for a "correlation matrix" that tells me the strength of association between each column item and each row item (much like the multi-trait multi-method approach used in some of the studies I'm looking at).
Well, let's think of what we have in a multitrait-multimethod matrix: A set of variables (usually scores on tests or scales), and the correlations between scores on each test and each other test. The results are shown in a table that looks a little like your crosstabs, yes, but a correlation matrix is quite a different thing to a table showing crosstabs. One clear (if a bit superficial) way they differ is that in a correlation matrix (e.g. an MTMM), the variables labelled in the top row are the same variables as shown in the leftmost column. In crosstabs this isn't the case. In fact, the row and column categories aren't variables at all - they're just categories (not categorical variables either).

I thought the extent to which individual cell values depart from chance expectation are a classic example of a measure of the degree to which row and column categories "correlate" (in sofar as the term is technically acceptable in dealing with categorical data).
We would not really call this correlation. As above, correlations show relationships between variables. "Mood category = happy" and "Mood class = surprised" aren't variables; the std. residual for this entry in the table isn't a correlation. (arguably semantics, but it gets to the heart of what we mean by 'correlation')

If the "happy photo"/"happy word" cell indicates a vast majority are correctly choosing this word, then related cell residuals are at least a broad measure of the "strength" of the match, compared to other cell correspondences.
Broadly, yes. But, I would think that the proportion of individuals describing the happy photo with the word happy is probably a more direct measure of what you're interested in. E.g. by the table in post #10, 64/95 = 67.7% of respondents described the happy photo with the word "happy". You can also calculate a 95% confidence interval for this proportion (about 57 to 76%). If you're that way inclined, you can even calculate a p value testing a null hypothesis that respondents match the emotional words to the faces purely randomly (by chance), in which case the expected proportion would be 1/6=0.167. The p value is very small (less than .0001), as you might expect. You could also rank the photos in terms of the proportion of respondents selecting the hypothesised emotional description for each photo.

This isn't a complete analytic strategy, but it's sort of getting there: I think that considering your questions in terms of proportions may help to bring some clarity.

Both you and Karibiner voice the same concerns, saying that there is some kind of disconnect between my research question and my (attempted) chi-square treatment, and even the basic principle of my treatment and the conception of 'correlation' here.
I think the disconnect with the idea of correlation is covered above. As far as chi-square goes... it does, sort of, get closer to what I think you're trying to do. It can allow you to reject the broad null hypothesis that people match emotional expressions to photos purely randomly. The problems I see with this are 1) The general implausibility of that null hypothesis, and 2) the main result of the chi-square test doesn't really get at the heart of your research question. It doesn't directly tell you which pictures are most associated with a particular emotional descriptions, except in an indirect and not particularly intuitive way, i.e. via the std residuals and so on.

This presumably isn't just antipathy towards the use of p-values in psychology, - you both know (and so does Jacob) that we're stuck with them, and can only emphasise the importance of effect size and get authors to report them more as part of their research to try and standardise things.
Originally Posted by Zenid
Where is this disconnect is confusing both you and cowboybear? Are you absolutely sure that it's not simply due to the area of research that I'm involved ("soft psychology") and the 'cultural' differences relating to the way stats are applied in a given area?
I'm a little too idealistic to be convinced we're stuck with p values, but no, that isn't the main issue here! And it definitely isn't about 'cultural' differences - I also do what one would probably have to call "soft psychology". In fact, I'm working on an article right now about the validation of a psychometric test using facial expressions as its response format. Not so dissimilar!

14. ## The Following User Says Thank You to CowboyBear For This Useful Post:

Zenid (02-03-2012)

15. ## Re: Correlational design using nominal/ordinal data

1) assess which pictures are most clearly matched with a single descriptor word.
2) identify which set of pictures, that is, which human face, most successfully displays the full range of expressions.
Is it correct to assume that for every picture there is 1 "correct" classification?
And all subjects in your study have rated all pictures?

In that case perhaps a logistic regression approach might be used. Dependent variable
would be correct classification (yes/no). Predictors would be: the subject which
does the ratings (say, 100 subjects), the person from which the picture was taken
(say, 10 persons), and the expression (i.e. 6 pictures from each person). So in the
example you'd have 6000 lines of data.

In SPSS, for example, the "generalized linear models" procedure could then be
used for the pairwise comparisons between pictures, or between persons, while
taking into account the dependencies between measures from the same subjects
(see https://www-304.ibm.com/support/docv...id=swg21480476, last 2 paragraphs).

Kind regards

K.

P.S.
Are you absolutely sure that it's not simply due to the area of research that I'm involved ("soft psychology") and the 'cultural' differences relating to the way stats are applied in a given area?
The way I apply stats is mostely influenced by psychologists like
Jacob Cohen ("The earth is round, p < 0.05" ; "Things I have
learned (so far)"), RP Abelson ("Statistics as a Principled Argument"),
RL Rosnow & R Rosenthal ("Statistical procedures and the justification of
knowledge in psychological science"), LV Hedges ("How hard is hard
science, how soft is soft science? The empirical cumulativeness of research"),
Gerd Gigerenzer ("Midless Statistics"; "The Null Ritual: What You Always
or Paul Meehl (diverse). So it's essentially a question of proper formulation
of research questions, finding the appropriate strategy for the data analysis,
and knowing how to interpret the results of statistical significance tests
correctly. Not of imaginary cultural differences between psychologists'
statistics and statistics.

16. ## The Following User Says Thank You to Karabiner For This Useful Post:

Zenid (02-03-2012)