student attitudes towards partially automated peer grading ... · • th4 year undergraduate course...

Student Attitudes towards ���Partially Automated Peer Grading ���

in Mechanical TA

Jessica Q. Dawson ���James R. Wright���

Kevin Leyton-Brown

BACKGROUNDCPSC 430 – Computers and Society

•  4th year undergraduate course –  focus on critical reasoning about social implications of

computational advances

•  focus on frequent short, frequent writing assignments –  effective way to teach writing skills [Seabrook et al 2005]

–  provides many opportunities to practice

•  students complete a 300-word essay every week –  Problem: inefficient and expensive for manual TA

marking –  Solution: peer grading

BACKGROUNDPeer Grading and Mechanical TA

•  in peer grading: students grade each others’ assignments

•  peer grading often negatively perceived by students –  tend to believe lower quality/less fair than TA grading

[Kaufman & Schunn 2011]

•  a solution: Mechanical TA (see companion poster for details.) –  software system for partially automated peer grading,

developed by CPSC 430 course staff –  TAs remain in the loop:

•  mark essays/reviews before students graduate to ‘independent’ •  for ‘independent’ students: manage appeals and spot-checks

–  results over 3 offerings found evidence that MTA helped improve student learning and grading ability [Wright et al. 2015]

MEASURING PERCEPTIONS OF MTAResearch Questions

•  what are the students’ perceptions of the peer grading? –  how do students perceive the quality, appropriateness,

fairness helpfulness and accuracy of: 1. reviews they gave their peers on their writing, 2. reviews they received from peers

. . . and how to did these compare to TA reviews?

–  how helpful was the calibration (built in practice reviewing) and peer grading and in MTA for learning?

Data Collection

•  End-of-term survey conducted in CPSC 430 (2014 W1) •  n = 76 (response rate 83%)

Reviews students gave their peers

The majority of students rated the reviews they wrote favorably with respect to each factor.

11%  

22%  

5%  

17%  

24%  

66%  

58%  

61%  

49%  

53%  

22%  

17%  

34%  

30%  

17%  

0%   20%   40%   60%   80%   100%  

were  accurate  with  respect  to  the  grading  rubric.  

would  have  been  helpful  when  they  wrote  future  essays.  

were  fair.  

were  appropriate  in  tone.  

were  of  high  quality.  

Q: The reviews I gave my peers on their writing . . .

Strongly  disagree   Disagree   Neither  agree  nor  disagree   Agree   Strongly  Agree  

RESULTS

Reviews students gave, compared to TAs

The majority of students the reviews they wrote were about the same as how a TA would have reviewed the same paper.

16%  

31%  

4%  

17%  

28%  

72%  

55%  

75%  

68%  

58%  

5%  

12%  

19%  

11%  

9%  

0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%  

accuracy,  with  repect  to  the  grading  rubric  

helpfulness,  for  wriHng  future  lectures  

fairness  

appropriateness  in  tone  

quality  

Q: Compared to a TA review, the _________ of the reviews that I gave my peers

Much  worse   Somewhat  worse   About  the  same   Somewhat  beKer   Much  beKer  

RESULTS

Reviews received from peers

The students’ perceptions of the reviews written by peers were more mixed.

7%  

12%  

7%  

11%  

31%  

28%  

24%  

18%  

28%  

35%  

26%  

43%  

36%  

38%  

22%  

28%  

18%  

36%  

16%  

5%  

5%  

8%  

8%  

7%  

0%   20%   40%   60%   80%   100%  

were  accurate  with  respect  to  the  grading  

were  helpful  when  I  wrote  future  essays  

were  fair.  

were  appropriate  in  tone.  

were  of  high  quality.  

Q: The reviews my peers gave me on my writing . . .

Strongly  disagree   Disagree   Neither  agree  nor  disagree   Agree   Strongly  Agree  

RESULTS

Reviews received, compared to TAs

For most factors, majority felt their peers’ reviews were worse than how a TA would have graded the same paper.

11%  

16%  

15%  

10%  

18%  

49%  

48%  

35%  

22%  

39%  

25%  

25%  

32%  

49%  

31%  

11%  

5%  

14%  

14%  

8%  

4%  

5%  

4%  

6%  

4%  

0%   20%   40%   60%   80%   100%  

accuracy,  with  repect  to  the  grading  rubric  

helpfulness,  for  wriHng  future  lectures  

fairness  

appropriateness  in  tone  

quality  

Q: Compared to a TA review, the __________ of the reviews I received were

Much  worse   Somewhat  worse   About  the  same   Somewhat  beKer   Much  beKer  

RESULTS

Helpfulness of peer reviewing for learning

Perceptions of helpfulness were mixed for activities asked about. For learning content and concepts, a small majority found peer review somewhat or very helpful (57%).

11%  

12%  

8%  

17%  

17%  

14%  

28%  

22%  

21%  

33%  

37%  

36%  

12%  

12%  

21%  

0%   20%   40%   60%   80%   100%  

Improving  WriHng  Ability  

Obtaining  large  amounts  of  feedback  

Learning  content  /  concepts  

Q:  Please  rate  the  extent  to  which  you  found  peer  reviewing  helpful  for  the  following  ac9vi9es:    

Very  unhelpful   Somewhat  unhelpful   Neither  helpful  nor  unhelpful   Somewhat  helpful   Very  helpful  

RESULTS

Helpfulness of calibration for learning

Majority of students found calibration helpful for all activities asked about – considered most helpful for activities specifically tied to reviewing.

5%  

4%  

5%  

5%  

18%  

16%  

25%  

32%  

55%  

43%  

48%  

48%  

20%  

35%  

19%  

12%  

0%   20%   40%   60%   80%   100%  

obtaining  a  lot  of  pracHce  reviewing  

becoming  an  independent  reviewer  more  quickly  

improving  the  quality  of  your  reviews  of  peers'  essays  

improving  the  quality  of  your  essays  

Q.  How  helpful  was  calibra9on  [built  in  prac9ce  reviews]  for  .  .  .  .  

Very  unhelpful   Somewhat  unhelpful   Neither  helpful  or  unhelpful   Somewhat  helpful   Very  helpful  

RESULTS

CONCLUSIONand possible next steps

•  Students feel positively about their own reviewing ability, perceived to be similar to TAs. –  calibration helpful as expected for learning how to review.

•  But many still doubt peers’ abilities. –  even though course staff also satisfied with reviewing ability. è how can we bridge this gap?

•  Perceived helpfulness of peer reviewing for learning and improving writing was mixed. –  could adjust types of feedback reviewers expected to

provide e.g., more qualitative and focused on writing skills.

References

•  Seabrook, R., Brown, G. D., Solity, J. (2005). Distributed and massed practice: from laboratory to classroom. Applied Cognitive Psychology, 19(1):107–122.

•  Kaufman J.H. & Schunn C.D. (2011). Students’ perceptions about peer assessment for writing: their origin and impact on revision work. Instructional Science, 3, 387–406.

•  Wright, J.R., Thorton C., Leyton-Brown, K. (2015). Mechanical TA: Partially Automatic High-Stakes Peer Grading. Proc. SIGSCE 2015, 96–101.