changing feedback – panel review session
TRANSCRIPT
5 July 2017: 07:00AM UTC/GMT Changing Feedback – Panel review session Justtoletyouknow:
Bypar'cipa'nginthewebinaryouacknowledgeandagreethat:Thesessionmayberecorded,includingvoiceandtextchatcommunica'ons(arecordingindicatorisshowninsidethewebinarroomwhenthisisthecase).Wemayreleaserecordingsfreelytothepublicwhichbecomepartofthepublicrecord.Wemayusesessionrecordingsforqualityimprovement,oraspartoffurtherresearchandpublica'ons.
Webinar Series
e-AssessmentSIG
Webinar Hosts ProfessorGeoffCrisp,PVCEduca'on,UniversityofNewSouthWalesg.crisp[at]unsw.edu.auDrMathewHillier,OfficeforLearning&Teaching,MonashUniversitymathew.hillier[at]monash.edu
Sessionchair:ProfessorSallyJordanOpenUniversityUK
Manchester, UK, 28-29 June 2017 About200delegatesfrom22countries.• Choiceof5masterclasses• 2keynotespeakers• 77researchpapers/prac'ceexchangesin10parallelsessions
• Posterandpitchsessions• ForTwiZerac'vitysee#AssessmentHEConf“Thefriendlyconference”
Conference themes • Exploringcontemporaryapproachestoassessmentandfeedbackinhighereduca'on.
• Enhancingassessmentandfeedbackatprogrammeandins'tu'onallevel.
• Cul'va'ngassessmentliteracy.• Integra'ngdigitaltoolsandtechnologiesforassessment.
• Developingacademicintegrityandacademicliteraciesthroughassessment.
• Assessmentforsocialjus'ce.
Keynote presentations
DrJanMcArthur,LancasterUniversityThedarkartsofassessment:FromSMARTtosocial
jus:ce
ProfessorDeniseWhitelock,TheOpenUniversityTechnologyenhancedassessment:Dowehaveawolf
insheep’sclothing?
Three research papers and practice exchanges to share with you…. CaroleSuIon,JaneCollingsandJoanneSellick,PlymouthUniversityModelsofExamina:onFeedbackJudyCohenandCatherineRobinson,UniversityofKentExploringtheeffectsofradicalchangetoassessmentand
feedbackprocesses:ApplyingTeam-basedlearningina
socialsciencemodule.LizAustenandCathyMalone,SheffieldHallamUniversityExploringstudentpercep:onsofeffec:vefeedback
ModelsofexaminaOonfeedback‘Notmanystudentswouldadmittoenjoyingtakingexamsbutifyou
wanttogetadegreethey’reanordealyouhavetosurvive.’
Guardian(2013)
JoSellick,CaroleSuIon,andJaneCollingsSchoolofLaw,CriminologyandGovernment
&TeachingandLearningSupport
Magne(2013)adaptedfromBrown(2007)
Feed-inWelldesignedassessment,briefing&prepara'on,DLEassessmentresourcesprac'cepapers&sessions,assessment&markingcriteria
Feed-forwardForma've‘feedback’,cues,discussion,mid-wayreviews,tutor,peer&self-reviewandfeedback
Feedbackendofexams,wriZen,verbal,mp3,YouTube,1:1,genericfeedback.construc'velegiblefeedback
Planningassessmentusingfeed-in,feed-forward&feedback
LearningSpace
ExaminaOonFeedbackProject
• Pre-surveystudentexperiencestogainstudentvoice.
• Literaturereviewinformedthedevelopmentofatoolkit.
• Stafftoolkitproduc'on• Post-surveystudentexperiences.
Outcomesoftheproject
1.ChangetotheUniversityofPlymouthAssessmentPolicy‘Receiveconstruc:vefeedbackaRerallassessments
includingexamina:ons.’hZps://www.plymouth.ac.uk/your-university/teaching-and-learning/guidance-and-resources/plymouth-university-assessment-policy-2014-20202.Examina'onstoolkitforstaff
– Genericfeedbackakerexams– 1:1feedbacksessionwithstudentswithexamina'onscripts
hZps://www.plymouth.ac.uk/uploads/produc'on/document/path/8/8424/University_of_Plymouth_Examina'on_Toolkit.docx
ExaminaOonFeedbackToolkit
KeyFactors• Examtype&purposeoftheexam(linkedtolearningoutcomes)
• CohortSize• Loca'on&typeofstudent• Timingand'mescales• Accesstoexamscripts• Withinamodule
– Forma'veandSumma'veassessmentapproach
ExamplesoffeedbacktoformaOveandsummaOveexaminaOonsandtests
SuggesOonshowto
GenericwriIen Summarycoveringminimumrequirementsandlinkedtolearningoutcomes–couldbebasedondis'nguishingbetweenclassifica'ons Email,
Moodle.‘Model’answersand/orexemplarsbankthatcanbeaccessedbystudentson-demandGenericverbal/audio Q&Asession Timetabledsession(note
poten'alofcontentcapture).Moodle.
Peerdiscussion/feedback(e.g.ofindividualforma'veand/orexemplars)Podcast/videoIndividualwriIen Template/summaryfeedbackformwithminimumrequirementsandlinkedtolearningoutcomeswithspecificreferenceonhowtoimprove/whattodonext
Hardcopy(collectedfrom/distributedbytutor/supportoffice).Moodle,PebblePad,Email
Individualverbal/audio 1:1appointmentwithexammarkerusingscript(forma'vemockand/orsumma've)
Generallypromotedwithon-demandappointmentsBlockedoutperiodsforappointments(‘examopenday’/‘feedbackweek’etc.).Moodle,PebblePad,Email
Mee'ngwithPersonalTutorwhohasreceivedreportsfrommarkersaspartofPDPprocessExamina'on‘openday’tolookthroughpaperswithtutorsIndividualaudiorecordingsuchaspodcast
Casestudy:SociologyProcessModule:Stage3:SocialandPoli'calTheory.AutumnSemesterAssessment:Seenexampaperessaystyle(2ques'onsin2hours)FormaOve
Feed-in:Dec/Jan:Examrelease,supportsessions,examansweroutlines
SummaOveFeedback:February:Groupfeedback,examscriptreview,invita'onfor1to1feedback
Impact
AcademicYear Avg.Exammark2008/09 53.84%2009/10 44.29%2010/11 49.50%2011/12 56.95%2012/13 54.16%2013/14 53.42%2014/15 63.33%2015/16 65.79%
Sociology:studentfeedback
– Understandingoftheassessedlearningoutcomes
– Expecta'onsandfocussedprepara'on
– Managinganxiety
– Enhancedunderstandingofmarksachieved
– Enhancedprepara'onleadingtoimprovedperformanceoverpreviousyears
– Staffwereinterestedinmyperformance
– Senseofachievement– Reduc'oninneedtoseekaddi'onalstafffeedback.
Feed-in Feedback
StaffChallenges
• Prac'calarrangements,including'metabling1:1slotsandschedulingforthere-sitexamina'on.
• Timemanagementandlargecohorts.• Tosensi'velybalanceexamscriptreturnwiththeneedforgradeprivacywhenconduc'nggroupfeedback.
• Toensurefeedbackisasusefulasfeed-in.• Establishingaclear,concretelinkbetweenexamfeedbackandimprovementsacrossacademicyears.
Emergingthemes• Examsarejustanotherformofassessmentthatstudentsshouldexpectfeedbacktounderstandandimprovetheirperformance.
• Logis'calconsidera'ons– Reten'onoftheexamscriptdoesnotpreventfeedback– Constrainedbyexam'metable– Requiresadap'onoffeedbacktomeetthe'meconstraints– Noequivalenceofe-submissionforexams– Challengesforfeedback,especiallySemester2
• Staffworkloadalloca'on– adjustmentstoaligntootherformsofassessment
Exploring the effects of radical change to assessment and feedback processes / Applying Team-based learning in a social science module
Judy Cohen and Catherine Robinson (KBS)
Funded through the Faculty Learning and Teaching Enhancement Fund
Outline of the presentation
• Economics module
• What is Team Based Learning?
• How was it implemented?
• Lessons learned
• Next steps
Economics for Business 2
Stage 2 module (109) Optional for B&M
Diverse student base
Poor attendance Poor engagement Mediocre marks
(Image source: http://www.mbacrystalball.com/blog/economics/macroeconomics/)
The UK’s European University
What is Team-Based Learning?
• More than group work
• A form of flipped learning
• Comprises of 3 key elements: • The Readiness Assurance Process (RAP) • Application exercises • Peer evaluation
• Applications need to meet the 4S criteria (Sibley and Ostafichuk (2014) • Significant; Same; Specific;
Simultaneous
The UK’s European University
Assessment for Learning
• Pre-reading
• Readiness Assurance Process
• Appeals
• Corrective Instruction
• 4 S application…
The UK’s European University
Implementation
Challenges – working with what we’ve got...
• Module choice • Institutional
constraints of Ø Physical space
Ø Module assessment
The UK’s European University
Implementation: How did we do it? • 110 students split into teams of 6/7
• Named after famous economists
• Lecture slots used to test the students’ readiness and the individual scores used as assessments
• Continual assessment at 5 points during the 12 week course
• Highest 3 scores were taken as their MCQ mark (worth 10% in total)
• This was administered using TurningPoint devices registered to individual students.
• 5 MCQs at the start, followed by team breakout time….
The UK’s European University
Was it worth it? 2015/2016 and 2016/2017
Attendance in seminars ↑ male 68.4% to 71%
↑ home students 69.3 to 79.8%
↓ female 80.4% to 75.3%
↓ overseas students 75.9% to 72.1
Performance in final exam ↑ Cohort average from 56.3% (standard deviation 12) to 63.0% in
2016/2017 (standard deviation 11.4)
↑ Anecdotal evidence of improved quality of writing
The UK’s European University
Student comments Viewpoint A Ø This module was taught differently
which I enjoyed. I enjoyed the team working element which promoted and encouraged learning and participation.
The UK’s European University
Viewpoint B Ø Never studied economics before, so layout of this module where you
read at home then come in and do test with no lectures was difficult for me. Difficult to understand terminology.
What I’d do differently
• More acknowledgement of the difference between novice and expert learners
• Changes to delivery • 2 hour workshop
• More on-line resources • KentPlayer
• Improvements to peer-evaluation system
The UK’s European University
Any questions?
Thank you
The UK’s European University
THE UK’S EUROPEAN UNIVERSITY
www.kent.ac.uk/kbs
Aim
Topresentselectedfindingsofamixedmethodsresearchprojectwhichanalysedthecharacteris'csoffeedbackthatstudentsvalued.
29
ResearchMethodology
Thistriangulatedstudentopinionsaboutfeedback(collectedthroughfocusgroups)withtheirevalua'onsoffeedbacktexts.Studentevalua'onsinformedcorpusconstruc'onandcontras'veanalysis.FocusGroups:5ac'vitybasedstudentfocusgroups2016.• n=28,4UGSocialSciencecourses,alllevels• suppor'ngstudentannota'ons• thema'ccodingofverbaldiscussions
StudentEvaluaOonsofFeedbackTexts• 95piecesofindividualwriZenfeedbackevaluated,
criteriaadaptedfromtheworkofNicol&MacFarlane-Dick(2007)
• textsmarkedtwice,rankedtocreateminicorporaofhighandlowrankingfeedback
30
A
B
9570
251
Findings
31
Length Achievement
Detail ForwardorientaOonInterpersonal
Praise
Error
Praise:Theimportanceofmanagingaffec'veneedsofthereader"IfIhadgotthatbackIwouldhavegone'Thetutorhatesme.Idon’twanttogoback’."• veryfewstudentscommentedonaposi'vetoneandthe
discussionfocusedonnega'vityandthedemo'va'ngimpactthishas
• feedbackwhichhighlightedwhatwaswrong,withoutprovidingsupportfordevelopment,wasfrequentlydiscussedasalivedexperience
• studentsdiscussedtheimportanceofreceivingpraise• praisewasseenasnecessary,evenonworkthatwasofa
lowstandard• clearcommentsaboutwhathasbeendonewellwas
neededtohighlightwhichaspectoftheworkcanberepeatedinforthcomingassessments
32
Ques'on:
Dostudentspreferlongerfeedback?Howlongistheop'mumfeedbackinyousubjectarea?
33
Length:Dostudentspreferlongerfeedback?
34
Thereisamoderatecorrela'on(0.51)betweenlengthoffeedbacktextandoverallstudentevalua'on• althoughbeZerscoringfeedbacktendstobelonger,thelongest
piecesoffeedbackwereevaluatednega'vely"Toolong!"Textlength399
• feedbacktextsbelow120wordsinlengthofplaintextaremorelikelytoelicitnega'vecomments"Tooshort,norealfeedbackgiven.Wouldn'tbeusefultome."Textlength66
• short/nofreetextcommentsviewedas"rushed”
Text CorpusA(top25texts) CorpusB(boIom25texts)TotalWordcount 4840 2943Averagetextlength 194 118Rangeoftextlengths 116-321 0-402Averagesentencelength 20 14Rangesentencelength 10-29words/sentence 0-20words/sentence
ComparisonOveralllengthCorpusA&B
Achievement:Whatisiden'fiedas'good'?
• 'good'appearswithsimilarfrequencyinbothcorpora• InCorpusAwhatisdescribedasgoodismuchmoredetailedandprecise.• 'good'isalsousedtolabeltopicdevelopment• CorpusBgooddescribessomethingfairlygeneric,followedbyimmediatecri'cism
35
A B
Goodintroduc'on.(x1)goodsummariesoverallgoodargumentsheregoodlinkstothecausesofcrime/theriots(x2)agoodclarifica'onofkeytermsgoodbreadthoftheorygoodacademiccri'queyoustartedtodothisneartheend,whichwasgood...Yourrecommenda'onswereunusuallyclear,whichisgoodbutneedmorediscussionWhatistheevidencethatthesearegood?
goodpoint(s)x7good(cri'cal)thinkingx3goodwork(x2)goodconclusion(x1)goodanswer(x1)good,but
• Whoappearsinyourfeedback?I/you/your.../we
36
Interpersonal:Whoisinthefeedback?You,I,we?
• comparedtoBri'shNa'onalCorpuswriZen(BNC6million
words)'you'and'your'appearwithveryhighkeynessfiguresacrossthewholesample.
• 'we'doesn'toccurineithercorpusAnalysis:• CorpusAismoreinterac'vethanB• introduc'onoftheselfintothetextmakesthefeedback
morebalanced• 'I'isusedtosokencri'calstatements
37
CorpusA(top25texts) CorpusB(boIom25texts)Rank Freq Keyness word Keyness Freq Rank1 130 570.3 your 282.7 70 35 163 381.1 you 142.6 78 5
ConclusionsStudentevalua'onsoffeedbackrevealeddis'nctdifferencesinlanguageusedinthetwocorpora.Theseneedtobeinterpretedinthelightofstudentcommentaryontheprocess.• Thisresearchhighlightstheimportanceoftheaffec've
responseofstudentreaderstofeedback,&howthismediatesstudentengagementwithfeedbackmessages.
• ItalsodrawsaZen'ontoins'tu'onalprocessesthatencourageproduc'onofobjec've,structuredandcriteriabasedfeedback.
• Differentaspectsoffeedbackallcontributetodiscursiveorienta'on,andagency,ofstudentreaderwithinthetext.
• Intermsofimplica'onsforprac'cethesefindingsshouldbeconsideredholis'cally,ratherthanasseparatecomponentswhichfunc'onindependently.
38
SampleuseofFeedbackForm.Thesecommentsfollowamarkinggrid.
Yourbestworkwaswhen:Providedacri:queofthetheore:calapplica:ontopar:cularriotresponses.
Showedadetailedknowledgeofriotresponses.
Toimprove,youneedtoworkon:-More‘analysis’relatedtocri:calthinkingaboutpolicy(relatedtopunishment)inbroaderterms.
-Dis:nguishingbetweencriminaljus:ceandsocialpolicydevelopmentsinrespectoftrendsin
punishmentresponses.
-Showamoredetailedandnuancedunderstandingoftheoryitself.Useitforanevenmorepowerful
cri:que.
-Accuratelyreferenceallonlinesourcese.g.placeforbooksandpagenumbersforar:cles.
Resourcesrecommendedtohelpyouimproveyourwork(withhyperlinkswhereapplicable):TheDefini:veReferencingGuideavailableviatheLibraryGateway
CTCMScreencastLectureonCri:callyAnalysingPolicy(videos1&2)
CTCMScreencastLectureonCri:callyAnalysingCaseStudiesandReports
CTCMScreencastLectureonCri:callyAnalysingTheory(videos1,2,3,4,5,6&7)
RemembertodiscussthisfeedbackwithyourAcademicAdvisortoassistyouracademicdevelopment☺
39
Example:Astyleguide
40
Do Don'tLength Writeatleast100words-freetext
forsumma'veassessmentWritelessthan100ormorethan300wordsforsumma'veassessment.
PosiOve Iden'fywhatthestudenthasdonewell.Findonethingyoulikeaboutthispieceofwri'ng.
Evaluatefirstyearworkbythirdyearstandards.Don'tqualifyeveryposi'vestatementyoumake-itreadsascri'cism.
Detailed Commentwithprecision.Makeacleartheconnec'onbetweenassessmentcriteriaandthissampleofstudentwri'ng.Unpackit,explainwhyit'seffec've.
Usegenericcommentsthatapplyacrossgradeboundaries.
Tone Beposi've,encouraging,andinclusive. Besarcas'c,oroverlydirec'veorharshlycri'cal.
ForwardorientaOon
Sumupyourresponsetotheirworkin2/3takehomemessagesthatcanbeappliedtotheirnexttask.Makeexplicittheconnec'onbetweenperformanceintext(poorrangeofsources)andunderlyingstudyprac'ces(needtoreadmore)-thismayrequirealotofmodelling.
Don'tfocusexclusivelyoninadequaciesofthelasttext,andhowthatcouldhavebeendonebeZer.Don'tleavethemtointerpretyourfeedbackordeducefromdetailedtextualcommentarywhattheyshoulddodifferentlynext'me.
Error Spellcheck Useacronyms,orobscureterms
Discussion CaroleSuIon,JaneCollingsandJoanneSellick,PlymouthUniversityModelsofExamina:onFeedbackJudyCohenandCatherineRobinson,UniversityofKentExploringtheeffectsofradicalchangetoassessmentand
feedbackprocesses:ApplyingTeam-basedlearningina
socialsciencemodule.LizAustenandCathyMalone,SheffieldHallamUniversityExploringstudentpercep:onsofeffec:vefeedback
Forthefuture,watchoutfor:SpecialissueofPrac::onerResearchinHigherEduca:on(PRHE)Journal.Onedayseminar,28thJune2018,Manchester:“TransformingAssessmentinHigherEduca:on:The
LearningPowerofFeedback”.
Keynotespeaker:ProfessorKaySambellNextAHEConference,June2019.AHEwebhZps://aheconference.com/
Webinar Series
Webinar Session feedback With thanks from your hosts
Professor Geoff Crisp, PVC Education, University of New South Wales g.crisp[at]unsw.edu.au
Dr Mathew Hillier, Office of the Vice-Provost Learning & Teaching Monash University mathew.hillier[at]monash.edu Recording available http://transformingassessment.com
e-AssessmentSIG