1

Using Peer Assessment & Technology to Support

Learning & Reflection

Fang Lou, Steve Bennett, Trevor Barker

School of Life Sciences/Computer Science

2

Introduction

• Overview of LTI project• Course in two schools (COM and LFS)– Level 4 BSc CS, Emedia Design,– Level 4 BSc Bio Sci HP & Sport FOHP

• Winning over difficult cohorts• Developing HOT skills• Supporting reflective learning• Improving performance

3

The technology• Electronic Voting Systems– CS: Used for peer marking of

previous cohort’s work– LFS: workshop for students’ opinion

• Google Spreadsheets:– Feedback in great detail 220 students x 3 marks sheets

containing 15 items in each (CS)• Optical mark sheet: – Save time to input marks (LFS)

• Online data collection: – Reflection and feedback from students (LFS)

CV

4

Peer Assessment

• Large amount of research papers on the beneficial effects of peer assessment on student motivation and ability to self assess

• Three comprehensive meta-studies: – Topping 1998, – Falchikov and Goldfinch 2000, – Zundert et al 2010

• Significant JISC projects REAP and PEER

5

The E Media Design Course

• A level 4 BSc course on the theory and practice of digital media

• Students didn’t generally apply the design theory to their creative artifacts

• Peer Evaluation of Previous Cohort’s work Introduced

• Produced an increase in student attainment but caused some student hostility

6

The Problem (or aftertaste)

25% MCQ 25% MCQ 50% Flash CV

25% MCQ25% EVSPeer MarkingExercise

50% Flash CV

2009 (Before Using Peer Assessment)

59% N=290

56% N=215

58% N=277

64.5% N=218

2010 (With Peer Assessment)

Learner 1: “(.. for example the marking criteria, it’s all over the place, how can we be tested on someone’s opinion??) so who knows.Learner 2: Maybe we will just guess what they are thinking.

I'm confused, how can a test be solely based on someone else opinion, this means even if you can justify why you've put a certain option it doesn't matter because your opinion doesn't mean anything.”

7

2011 Version: The measures

• Better marking proportions

• Better selection of exemplars– The six pieces of previous cohort work with the lowest

variance between markers• Rewriting of criteria– More detailed and graduated attainment descriptors

• Using a Live EVS session instead of QMP• Extremely detailed feedback on student marking

30% MCQ10% PeerMarkingEvent

60% Flash CV

8

Result

9

Result

10

Result

11

Fundamental Issue

• Does using peer-assessment help with the internalization of assignment criteria and expected standards?

• It seems so• But some students potentially merely regard it as

being asked to second guess the tutors.• There was far less controversy with the revised

approach• Students seemed to be buying in.

12

Some Issues• This result was the culmination of 3 years

research. We are not convinced that simply using without a great deal of additional work would necessarily be as effective.

• Setting up the sessions, writing and revising rubrics, selling the system to students and moderators, producing support lectures and materials etc. was not easy and took a great deal of time.

13

Issues• Student concerns had to be dealt with. • The EVS is often used in formative contexts – quiz

rather than assessment. (Does this devalue it?)• It is not absolutely reliable• It requires experience to:– Manage large EVS sessions (200+ students)– Write and reflect on rubrics and presentations– Collects and configure good samples for the sessions– Collect data– Set up the PPTs, software and hardware– Relate handset numbers to students’ names (not that easy)

14

Peer Assessment of a Full Lab Report- differences in two cohorts

• Level 4 BioScience (Bio) programme has been doing it for 5 years (Human Physiology module) – positive

• Level 4 Sports Science and Sports Therapy (Sports), introduced last year (Foundations of Human Physiology module) – quite negative

• The disparity between Bio and Sports • Challenge – winning over the SES/SPT cohort

15

Sequence of events

1. Laboratory study 2. Workshop after all students had completed the laboratory class – briefing3. Submission of the report4. Peer assessment (marking session):

clear marking criteria5. Appeal/reflection/feedback: face-to-

face? Email? Online?

16

Bio

EVS question – What do you think about peer assessment?

1. Glad to have a go2. Curious to find out

how it works3. Would prefer

lecturers to mark my work

4. Not comfortable with the responsibility

Sports

17

The problem

• Attitude – students did not see the point of doing it

• No incentive – don’t care marking well or not • Disappointment – when receive a carelessly

marked report• Results – lower level of satisfaction and huge

amount of complain and staff moderation

18

The measures

• Link to professionalism – selling the idea of peer assessment right from the Induction week; stress again in the workshop

• Reduce peer mark allocation from 20% to 10%• Allocate 5% for marking• Example of a report in the marking session• Moderate reports before releasing marks• Introduce the online feedback system (WATS) – 5% to

sports module

19

Findings• More positive attitude – EVS results• High engagement – many students

wrote a full page of comments• Raised satisfaction – reflection and

feedback results

Glad to have a go

Curious t

o find out how...

Would prefer le

cturers

..

Not comforta

ble with

th...

13%

4%

48%

35%

20

Return rates

FOHP

(Sports)HP

(Bio)

Number of students 81 273

Reports submitted 79 255Attended marking session 80 240

Online reflection 49 103

21

What do the students think?

SA A NAND D SD0.0

10.0

20.0

30.0

40.0

50.0

60.0

I benefited from being engaged with the marking criteria prior

to writing up the reportSA A NAND D SD

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

16.3

59.2

22.4

2.0 0.0

As a consequence of the peer assessment I

feel better prepared for my next lab report

Sports (FOHP)

Bio (HP)

22

the lab reports amongst peers also aided the learning process as you get another perspective.

The discussions it raised while marking the lab reports amongst peers also aided the learning process

I was surprised that it would help me

with my lab report and in the future

It is beneficial to do, however I do not think it needs to be done all

the time. Once or twice is enough to get a general idea of how the

marking works and how to improve your work.

It has made me reflect

more deeply than normal

I didn't find it useful as the report I was

marking hugely lacked in effort

23

Key points for success• Organisation of sessions– Making sure the technology works

• Marking criteria• Choosing Exemplars• Continuous improvement based on reflection• Selling the idea to students including briefing• Encouraging students: Reflection and

Feedback– Technology can help

24

Acknowledgements• LTI Support – Enhancement Awards 2011-12• Fang’s colleagues (Mike Roberts and Chris Benham)• References

– Barker, T. & Bennett, S., (2010), Marking Complex Assignments Using Peer Assessment with an Electronic Voting System and an Automated Feedback Tool, Proceedings of International Computer Assisted Assessment (CAA 2010),20-21 July, 2010, Southampton, UK.

– Barefoot, H., Lou, F. & Russell, M. (2011) Peer Assessment: Educationally Effective and Resource Efficient . Blended Learning in Practice, May, 2011

– Bennett, S. & Barker, T (2011a), Using Electronic Voting and Feedback: Developing HOT Skills in Learners,

presented at SOLSTICE 2011, June 8-9, Edge Hill University, UK – Bennett, S. & Barker, T (2011b), The Use of Electronic Voting to Encourage the Development of Higher Order

Thinking Skills in Learners. , Proceedings of International Computer Assisted Assessment (CAA 2011), July, 2011, Southampton, UK

– Lou, F., Barefoot, H., Bygate, D. and Russell, M. (2010) Using technology to make an existing assessment more

efficient. Poster at the International Blended Learning Conference. June, University of Hertfordshire, UK