writing motivating feedback professor denise whitelock the open university, walton hall, milton...

Writing motivating feedback

Professor Denise WhitelockThe Open University, Walton Hall,

Milton Keynes MK7 6AA, [email protected]

DMW_eAssessment which way now seminar_LKL_ May 2013

www.storiesabout.comwww.storiesabout.com/

[email protected]

What about emotional support in the feedback?

• Difficult at times to receive written feedback

• Not just a cognitive response

• How can we support tutors to give both cognitive and socio emotional feedback?



Coding the tutor comments

Categories Specific Examples

Positive Reactions

A1

A2

A3

1. Shows solidarity

2. Shows tension release

3. Shows agreement

Jokes, gives help, rewards others

Laughs, shows satisfaction

Understands, concurs, complies, passively accepts

Attempted Answers

B1

B2

B3

4. Gives suggestion

5. Gives opinion

6. Gives information

Directs, proposes, controls

Evaluates, analyses, expresses feelings or wishes

Orients, repeats, clarifies, confirms

Questions

C1

C2

C3

7. Asks for information

8. Asks for opinion

9. Asks for suggestion

Requests orientation, repetition, confirmation, clarification

Requests evaluation, analysis, expression of feeling or wishes

Requests directions, proposals

Negative Reactions

D1

D2

D3

10. Shows disagreement

11. Shows tension

12. Shows antagonism

Passively rejects, resorts to formality, withholds help

Asks for help, withdraws

Deflates others, defends or asserts self

Bales’ Interaction Process


Identifying trends: H801

0 5 10 15 20 25

A Pass 1

A Pass 2

A Pass 3

A Pass 4

B Pass 1

B Pass 2

B Pass 3

B Pass 4

C Pass 1

C Pass 2

C Pass 3

C Pass 4

D Pass 1

D Pass 2

D Pass 3

D Pass 4

Ba

les'

In

tera

cti

on

al

Ca

teg

ori

es

at

ea

ch

Pa

ssL

leve

l

Number ofIincidences

Graph to show conflated Bale’s categories against mean number of incidences in H801 scripts


Automating the coding: Open Mentor?• “An open source mentoring tool for tutors”

• “mentoring” = designed to help people learn how to give feedback effectively, through reflection and social networks

• “tutors” = primarily intended for teaching staff, but with clear applications for those involved in quality

• Comments match marks

• Socio emotive feedback acknowledged


‘A’ - Positive Reactions

Category Examples of Rules Examples of commentsA - Positive Reactions1. Shows solidarity A1 ...excellent... Excellent Conclusions.

A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.A1 ...nicely... Very nicely stated. Your analysis is thorough and your

conclusions consistent regarding the attractiveness of the budget airline sector. This is a good example of critical thinking.

A1 ...well presented... Very well presented diagram with interesting information.A1 ...effective use... Effective use of the case material here.A1 …well (structured|stated)… Report very well structured.A1 ...(well|clear)(ly)*

(structured|structure|summary| summarised|presented|presentation)...

The corporate vs. business unit strategy is well presented and nicely tied to strategies.

A1 ...reasonable.... A reasonable structure as listed in your table of contents.A1 ...useful point(s)... Generally useful points in this section.

2. Shows tension release A2 ... a helpful...

A2 …(thanks|thank you)…3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.

A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation, though it still has to control costs.

4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you analyse ??

OpenMentor Transfer: JISC funded

• JISC funded project

• Transfer OpenMentor technology to King’s and Southampton

• Our contact details, blog and references -http://omtetra.ecs.soton.ac.uk/wordpress


http://omtetra.ecs.soton.ac.uk/wordpress/



Praise for effort and not just ability

• Praise for ability per se can hinder learning (Mueller & Dweck, 1998)

• Praise = being clever

• Negative feedback now without ability

• Disempowering and demoralising


Mueller & Dweck (1998)

• Raven’s Matrices (IQ)

• First test pupils praise either for effort or ability

• Second test most difficult

• Third test medium difficulty. Score up 1 points for pupils praised for effort. Down 1 point ability


Open Comment addresses the problem of free text entry

• Automated formative assessment tool

• Free text entry for students

• Automated feedback and guidance

• Open questions, divergent assessment

• No marks awarded

• For use by Arts Faculty


Stages of analysis by computer of students’ free text entry for Open Comment: advice with respect to content (socio-emotional support stylised example)• STAGE 1a: DETECT ERRORS E.g. Incorrect dates,

facts. (Incorrect inferences and causality is dealt with below)

• Instead of concentrating on X, think about Y in order to answer this question Recognise effort (Dweck) and encourage to have another go

• You have done well to start answering this question but perhaps you misunderstood it. Instead of thinking about X which did not…….. Consider Y



Computer analysis continued

• STAGE 2a: REVEAL FIRST OMISSION

• Consider the role of Z in your answer Praise what is correct and point out what is missing Good but now consider the role X plays in your answer

• STAGE 2b: REVEAL SECOND OMISSION

• Consider the role of P in your answer Praise what is correct and point out what is missing Yes but also consider P. Would it have produced the same result if P is neglected?


Final stages of analysis• STAGE 3:REQUEST

CLARIFICATION OF KEY POINT 1

• STAGE 4:REQUEST FURTHER ANALYSIS OF KEY POINT 1(Stages 3 and 4 repeated with all the key points)

• STAGE 5:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS MISSING

• STAGE 6:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS NOT COMPLETE

• STAGE 7:CHECK THE CAUSALITY

• STAGE 8:REQUEST ALL THE CAUSAL FACTORS ARE WEIGHTED

Badge System: Mozilla


Where are we going?

• Opening up with Open Source

• Moving towards Advice for Action

• More work to do for Arts

• Open Comment - pedagogical model open to test

• Open Dialogues

• Automatic feedback that requires interaction

• Changing pedagogy

• Another handle on misconceptions


References• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J.

(2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium

• Whitelock, D., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P. & Saucedo, A. (2012) Supporting tutors with their feedback using OpenMentor in three different UK Universities. 10th International Conference on Computer Based Learning in Science, CBLIS 2012, Barcelona, Spain. 26-29 June 2012.

• Whitelock, D., Gilbert, L. & Gale, V. (2011) ‘Technology-Enhanced Assessment and Feedback: How is evidence-based literature informing practice?’ International Computer Assisted Assessment Conference, DeVere Grand Harbour Hotel, Southampton, 5/6 July 2011. http://caaconference.co.uk/wp-content/uploads/WhitelockB-CAA2011.pdf

• Whitelock, D. (2010) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342.


http://caaconference.co.uk/wp-content/uploads/WhitelockB-CAA2011.pdf

http://caaconference.co.uk/wp-content/uploads/WhitelockB-CAA2011.pdf

References (2)

• Whitelock, D. & Watt, S. (2008) ‘Putting Pedagogy in the driving seat with Open Comment: an open source formative assessment feedback and guidance tool for History Students.’ CAA Conference 2008, Loughborough University, 8/9 July 2008, edited by Farzana Khandia pp. 347-356 ISBN 0-9539572-7-6 http://kn.open.ac.uk/public/document.cfm?docid=11638

• Whitelock, D. & Watt, S. (2007) e-Assessment: How an we support tutors with their marking of electronically submitted assignments? Ad-Lib Journal for Continuing Liberal Adult Education, Issue 32, March 2007 pp 7-9, ISSN 1361-6323.


http://kn.open.ac.uk/public/document.cfm?docid=11638

References (3)

• Whitelock, D. (2006) Electronic Assessment: Marking, Monitoring and Mediating Learning. In McAndrew, P. and Jones, A. (eds) Interactions, Objects and Outcomes in learning. Special Issue of International Journal of Learning Technology. Vol. 2, Nos 2/3 pp 264-276.

• Whitelock, D. & Watt, S. (2006) OpenMentor: opening tutors eyes to the written support given to students in their assignments. JISC Conference 2006, Information & Communication Technology in Education and Research. International Conference Centre, Birmingham, 14 March 2006.


writing motivating feedback professor denise whitelock the open university, walton hall, milton...

Documents

weighted slide

point ability dmw

ukwordpress dmw

mozilla dmw

misconceptions dmw

demoralising dmw

answer stage

written feedback