The Quality of Peer-Feedback in the The Quality of Peer-Feedback in the Computerised Peer-Assessment of Essays? Computerised Peer-Assessment of Essays?

The case for awarding ‘marks for marking’ The case for awarding ‘marks for marking’ based upon peer feedback not marks.based upon peer feedback not marks.

Phil DaviesPhil Davies

School of ComputingSchool of Computing

OR

Lecturer getting out of doing marking

Need for assessment?Need for assessment?

• As tutors we are trying to “separate” As tutors we are trying to “separate” the sheep from the goats via the the sheep from the goats via the assessment process. assessment process.

• This can often be difficult with the This can often be difficult with the time constraints imposed on tutors, time constraints imposed on tutors, in what Chanock describes as “more in what Chanock describes as “more goat-friendly times” (Chanock, 2000). goat-friendly times” (Chanock, 2000).

• Feedback against time!Feedback against time!

Defining Peer-AssessmentDefining Peer-Assessment

• In describing the teacher ..In describing the teacher ..

A tall b******, so he was. A tall thin, mean A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. b******, with a baldy head like a lightbulb. He’d make us mark each other’s work, He’d make us mark each other’s work, then for every wrong mark we got, we’d then for every wrong mark we got, we’d get a thump. That way – he paused – ‘we get a thump. That way – he paused – ‘we were were implicated in each other’s painimplicated in each other’s pain’ ’

McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000,page (Pete McCarthy, 2000,page 68)68)

Why Peer-AssessmentWhy Peer-Assessment• Working together to learn (and be assessed)Working together to learn (and be assessed)

• PerceptionsPerceptions– Staff > Lecturer getting easy life not markingStaff > Lecturer getting easy life not marking– Student> Lecturer getting easy life not markingStudent> Lecturer getting easy life not marking

• Student Awareness of Benefits …. IMPORTANT Student Awareness of Benefits …. IMPORTANT TANGIBILITYTANGIBILITY

• Success dependant upon scalability Success dependant upon scalability (computerisation) .. “Student Numbers have (computerisation) .. “Student Numbers have risen dramatically since 1991 without a risen dramatically since 1991 without a concomitant increase in resources” (Pond et concomitant increase in resources” (Pond et al, 1995)al, 1995)

Computerised Peer-Computerised Peer-AssessmentAssessment• CAP SystemCAP System

• Permits students to mark & comment the Permits students to mark & comment the work of other students . (normally 6-8)work of other students . (normally 6-8)

• Also initial self-assess stage (reflection) … Also initial self-assess stage (reflection) … used as a used as a standard of expectationstandard of expectation

• Internet, not Web-based system Internet, not Web-based system (developed in Visual Basic / Access)(developed in Visual Basic / Access)

Having done the marking, what Having done the marking, what next?next?

• Students should receive feedbackStudents should receive feedback

• What feedback?What feedback?– Marks Marks – CommentsComments

• Which is most important?Which is most important?– To students or staffTo students or staff

AUTOMATICALLY EMAIL THE MARKER ..

ANONYMOUS

What should the marker do?What should the marker do?ReflectReflect

• Look at essay againLook at essay again

• Take into account the essay owner’s Take into account the essay owner’s commentscomments

• Further clarification (if it is needed, then is Further clarification (if it is needed, then is this a ‘black mark’ against the marker?)this a ‘black mark’ against the marker?)

• Try to ‘appease’ the essay owner?Try to ‘appease’ the essay owner?

• Modify mark based upon reflection?Modify mark based upon reflection?

• Give more feedbackGive more feedback

Exercise OneExercise One

• In the top box … what are the In the top box … what are the inherent advantages and inherent advantages and disadvantages of peer-assessmentdisadvantages of peer-assessment

• Middle Box … Mark (/10) and Middle Box … Mark (/10) and comment to supportcomment to support

• Bottom Box … Mark the marker /10 Bottom Box … Mark the marker /10 and comment to supportand comment to support

• Now give the work back!Now give the work back!

Must be rewarded for doing the Must be rewarded for doing the ‘mark for marking’ process .. ‘mark for marking’ process .. Based on qualityBased on quality

• How to judge?How to judge?

• Standard of expectation (self-assessment)Standard of expectation (self-assessment)

• Marking consistencyMarking consistency

• Commenting, quality, measure against markCommenting, quality, measure against mark

• Discussion ElementDiscussion Element

• Need for additional comments – black mark?Need for additional comments – black mark?

• Reaction to requests / further clarificationReaction to requests / further clarification

How easy to get an automated How easy to get an automated mark for marking?mark for marking?

• Statistically fairly easy to create a mark Statistically fairly easy to create a mark for marking based upon marksfor marking based upon marks

• Take into account high and low markersTake into account high and low markers

• Standard of expectationStandard of expectation

• Consistency … judge against final mark Consistency … judge against final mark awarded for an essay (compensated awarded for an essay (compensated median)median)

• What about the comments?What about the comments?

Feedback IndexFeedback Index

• Produce an index that reflects the Produce an index that reflects the quality of commentingquality of commenting

• Produce an average feedback index Produce an average feedback index for an essay (also compensated?)for an essay (also compensated?)

• Compare against marker in a similar Compare against marker in a similar manner to marks analysismanner to marks analysis

• Where does this feedback index come Where does this feedback index come from and is it valid?from and is it valid?

The way to get the feedback The way to get the feedback index?index?

• Develop an application??Develop an application??– C-Rater?C-Rater?

• Spelling MistakesSpelling Mistakes

• Similar Meanings?Similar Meanings?– That was coolThat was cool– Really ChocReally Choc– Really Good EssayReally Good Essay

• ManuallyManually

Commonality!!Commonality!!

• In the 67 essays that were markedIn the 67 essays that were marked– Only 96 commentsOnly 96 comments– 44% positive and 56% negative44% positive and 56% negative– Highly criticalHighly critical if something not explained properly if something not explained properly

(21% of total comments (of which 73% were (21% of total comments (of which 73% were negative)negative)

• Comments grouped into 10 categoriesComments grouped into 10 categories

• Need to QUANTIFY these comments .. Need to QUANTIFY these comments .. Feedback indexFeedback index

• Create a database holding positive & negative Create a database holding positive & negative (by category)(by category)

-7 -6 -5 -4 -3 -2 -1 -0 +0 +1 +2 +3 +4

34 34 51 56 52 56 61 63 70 64 68 70

39 52 61 46 57 58 62 72 65 70

44 45 52 54 66 59 62 69 68

34 35 57 66 65 65 66 69

56 59 68 64 63 71

61 59 66 66 66 66

55 48 60 69 68

61 57 58 65 73

59 61 60

63

62

68

34 37.8 45.8 59.6 55.6 58.8 62.9 62.8 67.5 68 68 70

General points of noteGeneral points of note

• Main criteria used for peer-markingMain criteria used for peer-marking– Explanations, conclusions, references, examplesExplanations, conclusions, references, examples

• Better students were critical (upper quartile)Better students were critical (upper quartile)– Better understanding permitting criticism?Better understanding permitting criticism?– Confidence?Confidence?– Hostility?Hostility?

• Weaker student in ‘cloud cuckoo’ land … Weaker student in ‘cloud cuckoo’ land … consistentlyconsistently

• Consistency:Consistency: 30-34, 35-39, 55-59; 65-69; 70- 30-34, 35-39, 55-59; 65-69; 70-7474

• Note 70-74 OVER-Comment & MarkNote 70-74 OVER-Comment & Mark

Time Consuming?Time Consuming?

• Can we formulate the marking processCan we formulate the marking process

• Take away need for quantification Take away need for quantification process of analyzing commentsprocess of analyzing comments

• Is it still peer-assessment if the Is it still peer-assessment if the students are told what to say?students are told what to say?

Exercise TwoExercise Two

1.1. I think you’ve missed out a big area of the researchI think you’ve missed out a big area of the research2.2. You’ve included a ‘big chunk’ that you haven’t citedYou’ve included a ‘big chunk’ that you haven’t cited3.3. There aren’t any examples given to help me There aren’t any examples given to help me

understandunderstand4.4. Grammatically it is not what it should be likeGrammatically it is not what it should be like5.5. Your spelling is atroceiousYour spelling is atroceious6.6. You haven’t explained anything to meYou haven’t explained anything to me7.7. You’ve directly copied my notes as your answer to You’ve directly copied my notes as your answer to

the questionthe question8.8. Most of what you’ve said is wrong Most of what you’ve said is wrong

STUDENT FRED

REFERENCES: Positive ………

Negative …….

Personal Valuation

5, 3, 2, 1

3, 1, 2

Is it my job to teach students Is it my job to teach students how to write essays, etc?how to write essays, etc?

• Assessment MUST be directedAssessment MUST be directed• Why bother writing essays, doing exam Why bother writing essays, doing exam

questions, etc. … doesn’t relate to needs questions, etc. … doesn’t relate to needs or learning outcomes of subjector learning outcomes of subject

• Post HND … N-tier … Assess the essays of Post HND … N-tier … Assess the essays of the final year (last year)the final year (last year)

• Preparation/Research: Judge knowledge Preparation/Research: Judge knowledge against last year’s results .. Both marks & against last year’s results .. Both marks & commentscomments

• Mistake!!Mistake!!

Can peer-assessment benefit Can peer-assessment benefit other subject areas?other subject areas?

• Java Programming with CoursemarkerJava Programming with Coursemarker• Stuart Lewis’ ideaStuart Lewis’ idea• Students create a solution to a Students create a solution to a

programming assignmentprogramming assignment• Submission(s)Submission(s)• Peer-Evaluate other solutionsPeer-Evaluate other solutions• Comments … Marks for Marking Comments … Marks for Marking

(weightings)(weightings)

CourseMarker Core

CMCM

File Storage System

Marking System

Evaluation System

Exercise Developm. System Student Exercise Environment

• assignments• exercises• notes• questions• test methods

• solution template

• marking scheme

• exercise setup

•submission

• edit• compile• link• run

feedbackand mark

• final mark

• position in class

• course statistics

• course statistics

• flagging-up of ‘problem cases’

immediate support

TEACHER STUDENT

• re-usability

• automated marking - fair - frees time

• plagiarism check

• steep learning curve

• difficult setup

(but it’s getting easier)

• immediate feedback

• fast support

• additional overheads

TEACHER STUDENTS

• UNIX (Linux), Windows, Mac, based all platforms

• Assessment of text I/O assignments only no marking of graphical output• remote student / teacher access distance learning, open all hours

Advantages / Disadvantages

FEATURES

Computer Ass

isted Teach

ing and Assess

ment

Modula-2

Java

C

comments / questions

PeerMarker ScreenPeerMarker Screen

Student while markingStudent while marking

• Exposure to different solutionsExposure to different solutions

• Development of critical evaluative Development of critical evaluative skillsskills

• Useful experience of reading code for Useful experience of reading code for future employment situationsfuture employment situations

• Plagiarism? … Good solution / No Plagiarism? … Good solution / No understandingunderstanding

Student while reviewing Student while reviewing feedback from peersfeedback from peers

• Range of subjective markingRange of subjective marking

• Confirmation of objective automated Confirmation of objective automated markingmarking

• Anonymous discussion between Anonymous discussion between marker and marked marker and marked

Current position Current position

• Test system workingTest system working

• Changes following beta test in Changes following beta test in progressprogress

• Plans to try sample study again (at a Plans to try sample study again (at a more convenient time, and with added more convenient time, and with added rewards!)rewards!)

• Employ 2Employ 2ndnd Placement Student Placement Student

• Graphical InterfaceGraphical Interface

Some Points Outstanding or Some Points Outstanding or Outstanding PointsOutstanding Points

• What should students do if they identify plagiarism?What should students do if they identify plagiarism?

• Is it ethical to get students to mark the work of their Is it ethical to get students to mark the work of their peers?peers?

• Is a computerised solution valid for all?Is a computerised solution valid for all?

• At what age / level can we trust the use of peer At what age / level can we trust the use of peer assessment?assessment?

• How do we assess the time required to perform the How do we assess the time required to perform the marking task?marking task?

• What split of the marks between creation & What split of the marks between creation & markingmarking