Writing Assessment Study - question about correlation

[James Libby]

Greetings,

I am a Writing & Rhetoric PhD student working on a CFP for an assessment journal.

Long story short, we have developed a student scorecard for writing (self-assessment) that we want to try to correlate to scoring on a more standardized writing rubric (see attached).

​

The student scorecard is on the left (with made-up scores) and the AWC rubric is on the bottom (with made-up scores).

How might we analyze the correlation between these measures? Pardon my inelegant way of asking these questions - but we are interested in correlations at the level of total score - as the scorecard totals go up or down, how does that effect the scores on the AWC (and vice versa)? Further, we are interested in seeing how each of these measures correlate to each other - that is, how does Pride (sense of accomplishment/affect) correlate with each of the categories on the AWC? I'm really lost as to how to think about this data, and we are struggling a bit with our CFP (which has been tentatively accepted by the journal) but they've asked us to provide more detail on the statistical measures we will use to investigate the correlation between the two metrics. Any help or thoughts would be immensely appreciated - including other, similar studies to look up and read (I am not afraid of doing my homework; just lost).

Please advise.

 

[Karabiner]

How large is your sample size?

Could you state your objectives and research questions more clearly (you probably communicated them to the journal)?

With kind regards

Karabiner

 

[James Libby]

How large is your sample size?

Could you state your objectives and research questions more clearly (you probably communicated them to the journal)?

With kind regards

Karabiner

Thank you so much for the reply.

We haven't determined the sample size yet - pending acceptance, we will recruit teachers to enact the experiment. If you have any recommendations about sample size, we'd love to hear it - obviously, we will get as many as possible (though the article is due March 1 and COVID hasn't made any of this easy).

We are trying to build out a student self-assessment scorecard for formative assessment. From the proposal:

In this article, we take a sociocultural and cognitive approach (Purcell-Gates et al., 2004, 2016) to the formative assessment of young learners’ writing proficiency by designing a student writing self-assessment for students to fill out with regard to their own writing assignments. The self-assessment -- a scorecard -- is a five-point scale measuring students’ perceptions of their text production appropriate for assessing elementary students’ participation in writing tasks that includes items related to sociocultural (e.g. the numbers of peers with whom they have talked to about their writing/ the number of times they talked with the teacher about a particular task), affective, (pride of accomplishment and task completion) and cognitive (genre specific items such as does a narrative have a beginning, middle, and end, and the use of particular writing strategies) dimensions of writing. Beyond serving as a tool of assessment, the items of the scorecard are designed to reinforce pedagogically sound interactions and activity.

Now, we also proposed to "correlate the results of the student scorecards with the AWC" so as to measure how student's assessment of their own work correlates with teacher's assessment. In other words, how accurately does the scorecard predict teacher's assessment of the same writing product?

Let me know what else I can provide - and thank you very, very much for your attention and help!

Regards,

JRL

 

[Karabiner]

It seems as if you could correlate the single items of your newly developed instrument,
as well as the total score, with the AWC total score (Spearman‘s Rank correlation
coefficients).

With kind regards

Karabiner

 

[James Libby]

It seems as if you could correlate the single items of your newly developed instrument,
as well as the total score, with the AWC total score (Spearman‘s Rank correlation
coefficients).

With kind regards

Karabiner

Thank you! After looking into this a bit, it looks like a perfect way to go about this task!

Thanks again!

 

[James Libby]

It feels like Spearman's Rank correlation coefficients is a great answer to my question, but I would welcome any other thoughts (should there happen to be any).

Thanks so much!

 

[Karabiner]

Ordinal logistic regression.

With kind regards

Karabiner