Peer feedback on a draft TMA answer:Is it usable? Is it used?

Mirabelle WalkerDepartment of Communication and Systems

Outline of talk

• The module context• Introduction to the research• Is the feedback from peers usable?• Is the feedback from peers used?

The module context

• T215 Communication and information technologies• Block 3 Communicating and collaborating• Small group project

– groups of up to 8 students• Each group uses a wiki to create a shared document• Overall topic

– Online communication and collaboration

Group work using the wiki

• Choice of sub-topics from the block– Online communities; web 2.0; virtual worlds; etc.

• Each student – drafts a wiki page on their sub-topic– gives feedback on two members’ drafts – improves their draft page based on the peer feedback

The mark scheme

• Students are marked individually on– their final wiki page (30 marks)– the peer feedback they provided (10 marks)– using the peer feedback they received (5 marks)

• The group as a whole is marked on– how well they collaborated (5 marks)

The criteria students learn to use – and tutors use for TMA feedback • Does the document meet the brief – exactly?• Is it factually accurate?• Is the structure appropriate for the audience, purpose

and medium?• Is the style appropriate for the audience, purpose and

medium?• Is the technical level appropriate for the audience,

purpose and medium?• Is the English correct?

Introduction to the research

The research questions

• What is the peer feedback like: how does it compare with feedback from tutors? is it usable?

• Did students pay attention to the feedback they received from peers?

Some relevant numbers

• 27 sample TMA 03s (one from each tutor)• Mean number of comments per tutor on the final wiki

page: 26.4• 48 examples of peer comments – limited to 250 words

and assessed• Mean number of comments per peer on the draft wiki

page: 12.3

Some general findings

• Peers all used their word allocation – but some were more discursive than others

• Surprisingly little overlap between what the two peers said (only 16 out of 255 (6.3%) comments that pointed out problems and 54 out of 313 (17.3%) positive comments) – it appeared as if the second peer to post deliberately tried to find different things to say

Is the feedback from peers usable?

Criteria percentagesCriteria types compared

0

5

10

15

20

25

30

35

Meet brief Fact acc Structure Style Tech level English Other

Perc

enta

ge

Tutors

Peers

Criteria percentages (without ‘Other’)

Criteria types compared (without 'Other')

0

5

10

15

20

25

30

35

Meet brief Fact acc Structure Style Tech level English

Perc

enta

ge

Tutors

Peers

Categories of comment from previous research

• Content• Skills development• Motivating• Demotivating• Reference to resource(s)• Reference to future work

Categories of comment for this research

• Content• Skills development• Motivating• Confirming• Demotivating• Reference to resource(s)• Reference to future work

What is a ‘comment’?

On the whole you’ve done a good job of meeting the brief, but you haven’t quite got the balance of topics right. You’ve spent too long on xxxx at the expense of yyyy.

What is a ‘comment’?

• On the whole you’ve done a good job of meeting the brief, motivating comment

• but you haven’t quite got the balance of topics right. You’ve spent too long on xxxx at the expense of yyyy. content comment

Category percentages

Categories compared

0

5

10

15

20

25

30

35

40

45

Content Skills development Motivating Confirm

Perc

enta

ge

Tutors

Peers

Were peers setting a lower standard than tutors (1)?• Comparison of comments made by peers and tutor for

any particular student showed that most of the difference in ‘praise’ and ‘confirm’ comments was because:– peers were being precise in addressing the sub-elements of

meeting brief and structure in a way that tutors weren’t, and this accounted for much of the difference (silence from a tutor often implied a sub-element was OK)

– peers nearly all made positive overall comments on the wiki page at the start and/or end of their feedback, whereas tutors often didn’t make any positive overall comments on the wiki page (even on the PT3)

– peers tended to confirm style and technical level more than did tutors.

Were peers setting a lower standard than tutors (2)?

Also• Students were told in the question not just to focus on

problems but to find positive things to say as well.So• In general most peers were not setting a noticeably

lower standard than tutors, and where they did it tended to be in one or two criteria, not across the board.

• In one or two cases the peers seemed to be setting a somewhat higher standard than the tutor!

‘Depth’ of comment

• Depth 1 – indicates e.g. ‘More needed here’; ‘Poor structure’, ‘Good’

• Depth 2 – corrects/amplifies e.g. ‘You needed to mention xxxx’; ‘Your answer would have had a better structure if you had …’, ‘A good introduction’

• Depth 3 – explains the correction/amplification e.g. ‘You needed to mention xxxx because …’; ‘Your answer would have had a better structure if you had … This is because …’, ‘This is a good introduction because you have …’

Depth percentages

Depths compared

0

10

20

30

40

50

60

70

80

Indicate Correct/amplify Explain

Perc

enta

ge

Tutors

Peers

Tone (when pointing out a problem)

• Tentative/suggesting• Factual• Exhortatory• Questioning

Tone percentages

Tones compared

0

10

20

30

40

50

60

70

80

Suggestion Statement Exhortation Question

Perc

enta

ge

TutorsPeers

Is the feedback from peers used?

Did students use the peer comments they received?

Of all the comments they received:• they responded thoroughly to 40%• they made some attempt to respond to 11%• they did not respond to 49%

What made a difference to whether a student used a peer comment?

Student response

0

1

2

3

4

5

6

7

8

9

0-19 20-39 40-59 60-79 80-100

Percentage of comments used

Num

ber o

f stu

dent

s

Did student’s ability make any difference to whether a student used a peer comment?

No!

Did criterion make any difference to whether a student used a peer comment?

Student response by criterion

0

10

20

30

40

50

60

70

Meet brief Fact acc Structure Style Tech level English

Perc

enta

ge

Did depth make any difference to whether a student used a peer comment?

Student response by depth

0

10

20

30

40

50

60

70

Indicate Correct Explain

Perc

enta

ge

Did tone make any difference to whether a student used a peer comment?

No!

Some conclusions• Most students have learned how to give feedback using

the criteria in the module.• The fact that peers gave a much higher proportion of

motivating and confirming comments than tutors does not in general imply peers were setting a lower standard.

• Some students made much more use of peers’ feedback than others.

• The tone of a peer’s comment and the ability level of the recipient made no difference to whether the comment was used.

• Students were somewhat more likely to use comments that were related to structure or were of depth ‘explain’.

Answers to the research questions

• Is the feedback from peers usable? In general, yes.

• Is the feedback from peers used? By some students, yes – particularly so if a comment is about structure or of depth ‘explain’.