creating the perfect storm: making peer assessment work in required courses

Creating the Perfect Storm: Creating the Perfect Storm: Making Peer Assessment Work Making Peer Assessment Work

in Required Courses in Required Courses

John MacLeanJohn MacLean

Department of English, ISUDepartment of English, ISU

[email protected]@ilstu.edu

Benefits of Peer AssessmentBenefits of Peer Assessment

Develops critical and analytical thinking Develops critical and analytical thinking

Reinforces the skills and content needed Reinforces the skills and content needed for the project being assessedfor the project being assessed

Increases and diversifies the audienceIncreases and diversifies the audience

Increases the amount (person-hours) of Increases the amount (person-hours) of evaluation of each student’s projectevaluation of each student’s project

Saves grading time, allowing teachers Saves grading time, allowing teachers time to engage students more strategically time to engage students more strategically

Problems with Peer AssessmentProblems with Peer Assessment

Is it valid, reliable, and fair?Is it valid, reliable, and fair?

It is time consuming. Is it worth it?It is time consuming. Is it worth it?

Practical obstacles among studentsPractical obstacles among students– Lack of motivation (esp. in required courses)Lack of motivation (esp. in required courses)– Desire to “help” friendsDesire to “help” friends– Ideas about “equality”Ideas about “equality”– Social pressure from peers Social pressure from peers – Confusion about the processConfusion about the process– Lack of understanding, expertise – feel unqualifiedLack of understanding, expertise – feel unqualified– Little justification provided for grades awardedLittle justification provided for grades awarded

Deeper “Problem” with Peer Deeper “Problem” with Peer AssessmentAssessment

Challenges culture of Challenges culture of traditional educationtraditional education– Decenters the teacherDecenters the teacher– Teacher is no longer Teacher is no longer

the “arbiter of good” the “arbiter of good” – Can be uncomfortable Can be uncomfortable

for both students and for both students and teachersteachers

Teacher tells students Teacher tells students what to do, how to do what to do, how to do itit

Students do itStudents do it

Teacher evaluates itTeacher evaluates it

The Perfect StormThe Perfect Storm

Building critical and analytical skills into Building critical and analytical skills into the whole projectthe whole project

Practical techniques to address obstaclesPractical techniques to address obstacles

Building critical and analytical skills Building critical and analytical skills into the whole projectinto the whole project

(RGS/CHAT approach)(RGS/CHAT approach)

Analyzing the genreAnalyzing the genre– Collect examples of a ______Collect examples of a ______– What is a (“good”) ______ like? What is a (“good”) ______ like? – Why is it that way? Who is Why is it that way? Who is

involved? Where does it go? involved? Where does it go? What does it do? …What does it do? …

Composing a text in that genreComposing a text in that genre

Self evaluation of my textSelf evaluation of my text– Does it match my analysis?Does it match my analysis?

Peer group assesses process and product

Teacher encourages careful, serious assessment

Practical techniques to address Practical techniques to address obstaclesobstacles

Lessening, diffusing social pressure on studentsLessening, diffusing social pressure on students– Making the assessment anonymous (for both Making the assessment anonymous (for both

assessors and authors)assessors and authors)– Doing the assessment across sections Doing the assessment across sections – Having students evaluate projects individually (for Having students evaluate projects individually (for

homework) before the group evaluates themhomework) before the group evaluates them

Encouraging learning from peers; creating Encouraging learning from peers; creating positive peer pressure to work seriouslypositive peer pressure to work seriously– Having students do the assessment in groups with a Having students do the assessment in groups with a

diversity of ability levelsdiversity of ability levels– Using Google docs collaboratively – grading group Using Google docs collaboratively – grading group

members can see each others’ comments on the members can see each others’ comments on the projects they are evaluating and respond to themprojects they are evaluating and respond to them


Facilitating and building better discernment in Facilitating and building better discernment in evaluationevaluation– Having students grade multiple projects with a range Having students grade multiple projects with a range

of qualityof quality– Using the words Amazing, Good, Adequate, Poor for Using the words Amazing, Good, Adequate, Poor for

assessment (not the letter grades A, B, C, D)assessment (not the letter grades A, B, C, D)– Requiring students to provide justification for ratings Requiring students to provide justification for ratings

awarded (annotations on the projects, brief summary awarded (annotations on the projects, brief summary notes on the grading charts, letters to the authors)notes on the grading charts, letters to the authors)

– Having students and groups rank the projects they Having students and groups rank the projects they grade (best to worst)grade (best to worst)


Supporting students in a difficult taskSupporting students in a difficult task– Developing (with students) clear criteria for the project Developing (with students) clear criteria for the project

and for the assessmentand for the assessment– Having a clear process for the assessment – step by Having a clear process for the assessment – step by

step instructions, yet not making the process too step instructions, yet not making the process too tedioustedious

– Giving students intuitive tools, e.g. individual and Giving students intuitive tools, e.g. individual and group evaluation chartsgroup evaluation charts

– Using a similar system / process repeatedly Using a similar system / process repeatedly throughout the semester throughout the semester

– Being transparent about my involvement in the Being transparent about my involvement in the process – how I will review student assessment (e.g. process – how I will review student assessment (e.g. “teacher use only” parts of forms) “teacher use only” parts of forms)


Reassuring students uncomfortable with peer Reassuring students uncomfortable with peer assessmentassessment– Explicitly mentioning the pedagogical value of peer Explicitly mentioning the pedagogical value of peer

assessment and discussing its validity and reliability assessment and discussing its validity and reliability – Emphasizing to students that I will review the projects Emphasizing to students that I will review the projects

and the evaluations and may, with good reason, and the evaluations and may, with good reason, tweak or change student ratingstweak or change student ratings

– Allowing students to appeal grades (with justification)Allowing students to appeal grades (with justification)

Providing extrinsic motivationProviding extrinsic motivation– Awarding a significantly weighted grade for the Awarding a significantly weighted grade for the

assessment work (based on quality and quantity of assessment work (based on quality and quantity of comments, charts, discussion, justification of ratings, comments, charts, discussion, justification of ratings, etc.)etc.)

Example ProjectExample Project

This I BelieveThis I Believe essay project essay project– ““In this short project we will practice the process of In this short project we will practice the process of

analyzing a genre, composing a text in that genre, analyzing a genre, composing a text in that genre, and evaluating our text against our analysis. While and evaluating our text against our analysis. While there are many ways to “do” this analysis and there are many ways to “do” this analysis and composing, for this project we will follow the process composing, for this project we will follow the process outlined in my Analyzing Genres and Composing outlined in my Analyzing Genres and Composing Texts document. The genre / text for this project is Texts document. The genre / text for this project is the the This I BelieveThis I Believe essay. See essay. See http://thisibelieve.org/””

– Involved individual work and review and discussion Involved individual work and review and discussion with other studentswith other students

– Criteria for each part (analysis, composing, self-Criteria for each part (analysis, composing, self-evaluation) were provided and discussedevaluation) were provided and discussed

Example Evaluation ProcessExample Evaluation Process

Students submitted the projects on Google DocsStudents submitted the projects on Google DocsBased on previous grades, I created grading Based on previous grades, I created grading groups of varied ability.groups of varied ability.I anonymized the projects and distributed them I anonymized the projects and distributed them to the grading groups using Google Docs.to the grading groups using Google Docs.I provided evaluation instructions and individual I provided evaluation instructions and individual and group evaluation forms (see PDFs)and group evaluation forms (see PDFs)Students evaluated the projects individually first Students evaluated the projects individually first (annotating the projects in Google Docs), then in (annotating the projects in Google Docs), then in their grading groupstheir grading groupsI reviewed and graded the individual and group I reviewed and graded the individual and group evaluations – which involved a brief review of evaluations – which involved a brief review of the projectsthe projects

Some Good ResultsSome Good Results

Overall, I found the evaluations to be “accurate” Overall, I found the evaluations to be “accurate” – the students mostly awarded the same grade – the students mostly awarded the same grade as I would (the average of the grades awarded as I would (the average of the grades awarded was 83%). I also noticed that there was a good was 83%). I also noticed that there was a good degree of agreement among the students.degree of agreement among the students.I was impressed at the analytical and critical I was impressed at the analytical and critical depth of the discussions during the group depth of the discussions during the group grading sessions.grading sessions.I was similarly impressed with the quality of the I was similarly impressed with the quality of the comments and letters written by the students to comments and letters written by the students to their peerstheir peersSpecific examples:Specific examples:

Examples of Student EvaluationExamples of Student Evaluation

Our group ranked your essay as adequate. We Our group ranked your essay as adequate. We based our decision on a few exemplary aspects of your based our decision on a few exemplary aspects of your essay and several of the lacking aspects of your essay. essay and several of the lacking aspects of your essay.

In part one, we feel like you could have gone more in In part one, we feel like you could have gone more in depth on the genre analysis, but you did supply a good depth on the genre analysis, but you did supply a good overall description of the TIB genre. It also would have overall description of the TIB genre. It also would have helped you to have more specific examples, perhaps helped you to have more specific examples, perhaps including examples from essays that you read. Referring including examples from essays that you read. Referring back to your statement that the genre should not include back to your statement that the genre should not include any exciting or enjoyable features, we thought the exact any exciting or enjoyable features, we thought the exact opposite. TIB first originated to inspire, and a way of doing opposite. TIB first originated to inspire, and a way of doing that is entertaining your audience.We did like how you that is entertaining your audience.We did like how you described the formality of the genre by saying that an described the formality of the genre by saying that an essay should not be written like a speech.essay should not be written like a speech.


In part 1 of your project, you supplied a good, In part 1 of your project, you supplied a good, concise, accurate description of the context of the concise, accurate description of the context of the genre. You included many examples and genre. You included many examples and explained how each one followed the genre. We explained how each one followed the genre. We liked that you included a precaution noting that the liked that you included a precaution noting that the writers weren’t restricted to the proposed writers weren’t restricted to the proposed guidelines. In general, it was a very well written guidelines. In general, it was a very well written and concise summary of the genre. It was and concise summary of the genre. It was enjoyable to read a fluid summary with good word enjoyable to read a fluid summary with good word choice in it. The only thing we can thought may choice in it. The only thing we can thought may improve the summary of the genre would be improve the summary of the genre would be including the history and trajectory of it.including the history and trajectory of it.


In part 2, we felt you did a very good job of In part 2, we felt you did a very good job of making your essay. We liked how you told making your essay. We liked how you told the story about the conversation you had the story about the conversation you had with your sister; you really explained when it with your sister; you really explained when it was exactly that you came to realize that was exactly that you came to realize that you had this belief. The only thing we felt you had this belief. The only thing we felt you should change is the part in the last you should change is the part in the last paragraph where you your belief being paragraph where you your belief being “wrong or ridiculous”. We thought that this “wrong or ridiculous”. We thought that this comment brought down the overall comment brought down the overall effectiveness of your essay. effectiveness of your essay.

DiscussionDiscussionIs it valid, reliable, and fair? (all citations below relate to Is it valid, reliable, and fair? (all citations below relate to English composition)English composition)– Huge discussion – What do they mean? Who decides? Do we Huge discussion – What do they mean? Who decides? Do we

want them? See, e.g. Colombini and McBride (2012) want them? See, e.g. Colombini and McBride (2012) – Straighter answer – Yes, reliable (with training) Bird (2011) - Straighter answer – Yes, reliable (with training) Bird (2011) -

76% correspondence between teacher rating and student rating 76% correspondence between teacher rating and student rating (on 7 point scale) (on 7 point scale)

– Yes, focus on the learning with community-based assessment Yes, focus on the learning with community-based assessment (and rubric development) (Inoue, 2004; Elbow, 2006)(and rubric development) (Inoue, 2004; Elbow, 2006)

– My experience – yes, but not for everything. I was less My experience – yes, but not for everything. I was less impressed with the group peer grading with a project that was impressed with the group peer grading with a project that was more sophisticated and involved more new and varied content. more sophisticated and involved more new and varied content. (APGdb project)(APGdb project)

Bird, R. (2011). Peer Evaluation in the Composition Classroom. Presented at the Conference on College Composition and Communication, Atlanta, GA.Colombini, C. B., & McBride, M. (2012). “Storming and norming”: Exploring the value of group development models in addressing conflict in communal

writing assessment. Assessing Writing, 17(4), 191–207. doi:10.1016/j.asw.2012.05.002Elbow, P. (2006). Do we need a single standard of value for institutional assessment? An essay response to Asao Inoue’s “community-based

assessment pedagogy”. Assessing Writing, 11(2), 81–99. doi:10.1016/j.asw.2006.07.003Inoue, A. B. (2004). Community-based assessment pedagogy. Assessing Writing, 9(3), 208–238. doi:10.1016/j.asw.2004.12.001

DiscussionDiscussionIt is time consuming. Is it worth it?It is time consuming. Is it worth it?– More class time, but develops more More class time, but develops more

versatile skill set (when used appropriately)versatile skill set (when used appropriately)

Practical obstacles are reduced with Practical obstacles are reduced with smart, careful implementationsmart, careful implementation

Educational culture? Instead of Educational culture? Instead of either/or, how about both/and?either/or, how about both/and?

Your comments, ideas …Your comments, ideas …

Thank you!Thank you!

Further comments, ideas?Further comments, ideas?

[email protected]@ilstu.edu

creating the perfect storm: making peer assessment work in required courses

Documents

obstaclessupporting

evaluationhaving students

authorshaving students

seriouslyhaving students

drequiring students

peer group

peer assessment work

students clear criteria