3 June 2020: 07:00AM UTC e-Assessment Awards: Selected finalists showcase Session chair: Martha Gibson Edinburgh Business School, Heriot-Watt University.
Just to let you know: By participating in the webinar you acknowledge and agree that: The session may be recorded, including voice and text chat communications (a recording indicator is shown inside the webinar room when this is the case). We may release recordings freely to the public which become part of the public record. We may use session recordings for quality improvement, or as part of further research and publications.
e-Assessment SIG
Webinar Hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com
Webinar Series
@transformassess
Casestudiesfrom2020finalists
MarthaGibsonSponsorshipSecretary&Execu3veBoardMember.e-AssessmentManager,Heriot-WaAUniversity,UK.TwiAer:@MG_EBSLinkedIn:www.linkedin.com/in/martha-gibson123
TheeAAisaprofessionalbodythatpromotesbeAerassessmentofknowledge,skillsandcapabili3esofpeoplethroughtechnology.Itsmembersrepresentthewholeofthesector,fromacademicsandins3tu3onstotechnologyprovidersandlargecorporates.
1833IndividualMembers(freesubscrip3on)14BoardMembers(voluntary)20SponsorOrganisa3ons2326followersonTwiAer762membersoftheLinkedInDiscussionGroup
Podcasts,AnnualConference,events,papers,SIGs,newsleAers,casestudies,blogs,glossary,eventcalendar,socialmedia,plusthee-AAAwards!
hAps://e-Assessment.com
Thee-AssessmentAssocia3on
OneofourkeygoalsasanAssocia3onistocommunicatetheposi3vecontribu3onsthattechnologymakestoallformsofassessmentandweaimtoshowcasetheverybestprac3ce,researchandinnova3ontakingplace.
-Worldwideapplica3onsin2020fromCanada,USA,SaudiArabia,Scandinavia,UKandIreland.-7categories.WinnerstobeannouncedviaouronlineAwardsShownextweekandonTwiAer@eAssess#eAAwards-Separateandindependentjudgingpanelsforeachcategory.-En3reentryprocess,includingtheinterviewsofallshortlistedentrants,takesplaceonline.-2021AwardswillbeopenforentriesfromOctober2020.
hAps://eAssessmentAwards.com
Thee-AssessmentAwards
BestUseofSumma3veAssessmentBestUseofForma3veAssessmentBestResearchExcellenceinExportBestTransforma3onalProjectInnova3onLife3meContribu3onFulllistoffinalists:hAps://eassessmentawards.com/finalists-2020/
e-AssessmentAwardscategories
2020finalistswithustoday…BestUseofSumma@veAssessmentUniversityofReading(UK)forStudentProgressDashboard.Presenter:ProfessorEmmaMayhew
BestUseofSumma@veAssessmentNewcastleUniversity(UK)forNewcastleUniversityDigitalExamService.Presenters:JackEnnisandRebeccaGill
Innova@onCentreforAssessmentResearch,PolicyandPrac3ceinEduca3on(CARPE)(Ireland).Presenter:DrAnastasiosKarakolidis
Finalist:BestUseofSumma@veAssessment
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT Copyright University of Reading
Professor Emma Mayhew, University of Reading
THE IMPACT OF A NEW STUDENT ASSESSMENT DASHBOARD ON STUDENT LEARNING
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
DASHBOARD-MAIN PAGE
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
DASHBOARD-MAIN PAGE
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
DASHBOARD-MODULAR VIEW
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
CLICK THROUGH INTO A SPECIFIC MODULE
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
CLICK THROUGH INTO INDIVIDUAL ASSESSMENT
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
STUDENT RESPONSE
58,000+
Views since mid-November 2019
Staff Views, 7.0%
Student Views, 93.0%
When asked to give a high, medium or low value rating for each feature, the weighted running part average received the most ‘high’ ratings
A survey of 218 students showed that: 94% liked it 85% were more satisfied with their assessment experience 94% felt it would help them self-reflect on progress 95% were taking action to change assessment behaviours as a direct result 53% said they would look again at feedback 62% said it would improve tutorial conversation
High value ratings
LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT
Comments included “It is awesome!” “I clearly know where I stand and what I need to improve” “I can clearly see which module requires more study and improvement” “Satisfying to know where I am excelling” “It is motivational to keep work up to the same standard” “It has made me relax a bit” “It makes me worry less” “I have heightened my expectations of myself as I can see that with a little more work I can push my marks up”
STUDENT RESPONSE
Finalist:BestUseofSumma@veAssessment
Newcastle University Digital Exam Service
3/6/20 LearningandTeachingDevelopmentService 20
Jack Ennis!
Rebecca Gill!
Learning and Teaching Development Service !
Newcastle University!!
Current and planned provision
KeyObjec3ve:Makeonlinesumma3veassessmentpossibleforawiderangeofassessmenttypesandinnon-clusterenvironments
3/6/20 LearningandTeachingDevelopmentService 21
050
100150200
2007-8
2008-9
2009-10
2010-11
2011-12
2012-13
2013-14
2014-15
2015-16
2016-17
2017-18
2018-19
Numberofdigitalexamsdelivered
Academicunits
StudentWellbeing
ExamsOffice LTDS
ITServiceEstates
Transforming Assessment
JISC'TheFutureofAssessment'report(Spring2020)Technologycanbeusedtotransformassessmenttomakeitmore:Authen3cAccessibleAppropriatelyautomatedCon3nuousSecure
JISC,2020hAps://www.jisc.ac.uk/reports/the-future-of-assessment
3/6/20 LearningandTeachingDevelopmentService 22
Evaluation: digital written exams
3/6/20 LearningandTeachingDevelopmentService 23
24%
27%25%
17%
7%
HowdidyourexperienceoftakingthewriPenpartoftheexamoncomputer
comparetoyourpreviousexperiencesoftakingwriPenexamsonpaper?
MuchbeAer
SomewhatbeAer
NeitherbeAernorworse
Somewhatworse
MuchworseStudentevalua3onsofalldigitalwriAenexamstoMarch2020
itwassomucheasiertoeditmywork.Wri3ngisacyclicalprocess,sobeingabletogobackandamendmypreviousparagraphwasreallyhelpful
Evaluation: Bring Your Own Device (BYOD)
LearningandTeachingDevelopmentService 243/6/20
27%
31%
28%
11%3%
HowdidyourexperienceoftakingtheexamonyourlaptopcomparetoyourpreviousexperienceoftakinganonlineexamonaUniversitycomputerina
cluster?
MuchbeAer
SomewhatbeAer
NeitherbeAernorworse
Somewhatworse
MuchworseStudentevalua3onsofallBYODdigitalexamstoMarch2020
BeingabletotypeonakeyboardIamusedtowasgreat
Itwasmucheasiertoconcentrateonlaptopsastherewasnotasmuchofanoisecreatedfrompeopletyping
Evaluation: colleagues’ perspectives
3/6/20 LearningandTeachingDevelopmentService 25
allowedmetoprovideanexamthatiscustomisabletomymoduleinordertomeetmylearningoutcomes[…]Icanassessstudentsonarangeofskillsandknowledge-basedoutcomes. DrAdrianSmall,Opera3onsManagementmoduleleader
Ibelievethattheeaseofreadingeachpaperactuallyledtofairermarking.Studentswithpoorhandwri3ng[…]areomengiventhebenefitofdoubtandthisbiasdisappeared.
DrGabrieleJordanPsychologymoduleleader
allowedmetouseanexternaltool,andIwasabletoaskapplica3onques3onsmuchmorecomplexandinteres3ngasaresult.
DrCharlesMorissetCompu3ngmoduleleader
Opportunities and challenges
• Examroomtechnicalsupport• Logis3csatscale• Assis3vesomwareforlockeddownexams
• Incorpora3ngdiagramsdrawnonpaper
• Demandfornewtypesofdigitalexams
• Covid-19response
3/6/20 LearningandTeachingDevelopmentService 26
Finalist:Innova@on
Anastasios Karakolidis [email protected]
Using Animations in Assessment
The reality
Thevastmajorityoftestsinusetodayrelyheavilyonsta3ctext
The problem with text-based assessments Limitedintermsofthecomplexityofwhatcanbepresentedass3muliand/orresponses[1]
Test-takersarerequiredtohaveskills(e.g.,readingcomprehension)thatmaybeirrelevanttotheconstructthattheassessmentpurportstocapture[1,2]
Suchequityandfairnessissuesmightaffect:• test-takerswithpoorlanguageandreadingskills• non-na3vespeakers• peoplewithdyslexia
Construct-irrelevantvarianceàvalidity[3]
This innovation…
explorestheextenttowhichanimatedvideosareausefulalterna3veto
wriAentextintes3ng
Why animated videos?
Advantages:• replacemuchofwriAentext• enablethevisualisa3onofcomplexinforma3onthatcannotbeeasilycommunicatedviatext
• moreeffec3vethanothervideo-basedmediaChallenges:
• developmentcomplexi3es• cost
Development of an animated test Selec3onofasitua3onaljudgementtest(SJT)designedbyStevenStemler,RobertSternbergandtheircolleaguesatYaletomeasureteachers’prac3calknowledgeofhowtodealwithchallengingsocialsitua3onsatwork[4,5]
Collabora3onwithananima3oncompanytocreate15animatedvideosbasedontheoriginaltext-basedscenarios
A sample text-based scenario
The animated SJT
AnimaAonsdevelopedbyAnimatedscenarios.com-PlaEormdesignedbyPsycholate
The animated test
AnimaAonsdevelopedbyAnimatedscenarios.com-PlaEormdesignedbyPsycholate
The animated test
Process of animating 1s
t dram
2nd d
ram
Finaldram
The research study Experimentalcomparisonoftheanimatedvs.thetext-basedversionofthetest
Sample:na3ve(n=51)andnon-na3ve(n=78)English-speakingpre-serviceteachers-randomassignment
Measures:• SJTperformance• Englishproficiency• readingcomprehensionabilityinEnglish
• ThevarianceaAributedtoconstruct-irrelevantfactorswasreducedby9.5%
• Inotherwords,theuseofanimatedvideosreducedthedependencyoftestperformanceonirrelevantfactors,suchaspar3cipants’proficiencyandreadingcomprehensioninEnglish
Research question #1: What impact does the use of animated videos have on construct-irrelevant variance attributed to language and reading skills?
Research question #2: What impact does the use of animated videos have on test-takers’ reactions to the test?
Theanimatedversionofthetest
wasperceivedasmorevalidandfair(d=0.38)
wasperceivedasmoreenjoyable(d=0.50)
wasfoundtouselanguagethatwaslessdifficulttounderstand(η2=0.043)
So what? • Didanima3onsmakeasubstan3aldifference?• Weshouldcareaboutwhattest-takersthinkoftheteststheytake!
• Test-takerspercep3onsofatestcanhaveanimpacton:• theirastudestowardsthebodyorganisingtheassessment[1]• theirengagementwiththeassessment[6,7]• theirperformance[6,7]
• Isitworththeeffortandmoney?
The potential • Improvingequityandfairness
• Improvingstandardisa3oninassessment
• Tes3ngacrossdifferentpopula3ons
• Tes3nghard-to-measureconstructs
• Professionaldevelopment
References [1]E.C.Popp,K.Tuzinski,andM.Fetzer,“Actororavatar?Considera3onsinselec3ngappropriateformatsforassessmentcontent,”in
TechnologyandTes3ng:ImprovingEduca3onalandPsychologicalMeasurement,F.Drasgow,Ed.NewYork,NY:Routledge,2016,pp.79–103.
[2]J.Abedi,“Linguis3cfactorsintheassessmentofEnglishlanguagelearners,”inTheSAGEHandbookofMeasurement,G.Walford,E.Tucker,andM.Viswanathan,Eds.California,Ca:SAGE,2010,pp.129–105.
[3]AmericanEduca3onalResearchAssocia3on(AERA),AmericanPsychologicalAssocia3on(APA),andNa3onalCouncilonMeasurementinEduca3on(NCME),StandardsforEduca3onalandPsychologicalTes3ng.Washington,DC:AmericanEduca3onalResearchAssocia3on,2014.
[4]S.E.Stemler,J.G.EllioA,E.L.Grigorenko,andR.J.Sternberg,“There’smoretoteachingthaninstruc3on:Sevenstrategiesfordealingwiththeprac3calsideofteaching,”Educ.Stud.,vol.32,no.1,pp.101–118,2006.
[5]J.G.EllioA,S.E.Stemler,R.J.Sternberg,E.L.Grigorenko,andN.Hoffman,“Thesociallyskilledteacherandthedevelopmentoftacitknowledge,”Br.Educ.Res.J.,vol.37,no.1,pp.83–103,Feb.2011.
[6]Chan,D.,SchmiA,N.,DeShon,R.P.,Clause,C.S.,&Delbridge,K.(1997).Reac3onstocogni3veabilitytests:Therela3onshipsbetweenrace,testperformance,facevaliditypercep3ons,andtest-takingmo3va3on.JournalofAppliedPsychology,82(2),300–310.
[7]ScoA,J.C.,&Mead,A.D.(2011).Founda3onsformeasurement.InN.Tippins&S.Adler(Eds.),Technolgy-EnhancedAssessmentofTalent(pp.21–65).SanFrancisco,CA:Jossey-Bass.
Emailaddress:[email protected]
e-AssessmentAssocia@on
JointheeAAforfreeatwww.e-assessment.com/registerJointheeAALinkedIngroupandaddtothediscussionsFollowusontwiAer@eAssessSpreadthewordaboutthe2021Awardsprogramme.Findoutmoreatwww.eassessmentawards.comGetintouch!Sendinyourblogposts,thoughts,queries,[email protected]
Thankyouforlistening
Webinar Session feedback With thanks from your hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com Recording available http://transformingassessment.com
Next session 15 July 2020 Assessment in
Higher Education Conference
Preview for 2021 Register
http://taw.fi /15jul2020
Webinar Series
e-Assessment SIG
@transformassess