ta webinar 3 june 2020 slides2transformingassessment.com/sites/default/files/files/ta...3 june 2020:...

3 June 2020: 07:00AM UTC e-Assessment Awards: Selected finalists showcase Session chair: Martha Gibson Edinburgh Business School, Heriot-Watt University.

Just to let you know: By participating in the webinar you acknowledge and agree that: The session may be recorded, including voice and text chat communications (a recording indicator is shown inside the webinar room when this is the case). We may release recordings freely to the public which become part of the public record. We may use session recordings for quality improvement, or as part of further research and publications.

e-Assessment SIG

Webinar Hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com

Webinar Series

@transformassess

Casestudiesfrom2020finalists

MarthaGibsonSponsorshipSecretary&Execu3veBoardMember.e-AssessmentManager,Heriot-WaAUniversity,UK.TwiAer:@MG_EBSLinkedIn:www.linkedin.com/in/martha-gibson123

TheeAAisaprofessionalbodythatpromotesbeAerassessmentofknowledge,skillsandcapabili3esofpeoplethroughtechnology.Itsmembersrepresentthewholeofthesector,fromacademicsandins3tu3onstotechnologyprovidersandlargecorporates.

1833IndividualMembers(freesubscrip3on)14BoardMembers(voluntary)20SponsorOrganisa3ons2326followersonTwiAer762membersoftheLinkedInDiscussionGroup

Podcasts,AnnualConference,events,papers,SIGs,newsleAers,casestudies,blogs,glossary,eventcalendar,socialmedia,plusthee-AAAwards!

hAps://e-Assessment.com

Thee-AssessmentAssocia3on

OneofourkeygoalsasanAssocia3onistocommunicatetheposi3vecontribu3onsthattechnologymakestoallformsofassessmentandweaimtoshowcasetheverybestprac3ce,researchandinnova3ontakingplace.

-Worldwideapplica3onsin2020fromCanada,USA,SaudiArabia,Scandinavia,UKandIreland.-7categories.WinnerstobeannouncedviaouronlineAwardsShownextweekandonTwiAer@eAssess#eAAwards-Separateandindependentjudgingpanelsforeachcategory.-En3reentryprocess,includingtheinterviewsofallshortlistedentrants,takesplaceonline.-2021AwardswillbeopenforentriesfromOctober2020.

hAps://eAssessmentAwards.com

Thee-AssessmentAwards

BestUseofSumma3veAssessmentBestUseofForma3veAssessmentBestResearchExcellenceinExportBestTransforma3onalProjectInnova3onLife3meContribu3onFulllistoffinalists:hAps://eassessmentawards.com/finalists-2020/

e-AssessmentAwardscategories

2020finalistswithustoday…BestUseofSumma@veAssessmentUniversityofReading(UK)forStudentProgressDashboard.Presenter:ProfessorEmmaMayhew

BestUseofSumma@veAssessmentNewcastleUniversity(UK)forNewcastleUniversityDigitalExamService.Presenters:JackEnnisandRebeccaGill

Innova@onCentreforAssessmentResearch,PolicyandPrac3ceinEduca3on(CARPE)(Ireland).Presenter:DrAnastasiosKarakolidis

Finalist:BestUseofSumma@veAssessment

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT Copyright University of Reading

Professor Emma Mayhew, University of Reading

THE IMPACT OF A NEW STUDENT ASSESSMENT DASHBOARD ON STUDENT LEARNING

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

DASHBOARD-MAIN PAGE

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

DASHBOARD-MAIN PAGE

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

DASHBOARD-MODULAR VIEW

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

CLICK THROUGH INTO A SPECIFIC MODULE

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

CLICK THROUGH INTO INDIVIDUAL ASSESSMENT

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

STUDENT RESPONSE

58,000+

Views since mid-November 2019

Staff Views, 7.0%

Student Views, 93.0%

When asked to give a high, medium or low value rating for each feature, the weighted running part average received the most ‘high’ ratings

A survey of 218 students showed that: 94% liked it 85% were more satisfied with their assessment experience 94% felt it would help them self-reflect on progress 95% were taking action to change assessment behaviours as a direct result 53% said they would look again at feedback 62% said it would improve tutorial conversation

High value ratings

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

Comments included “It is awesome!” “I clearly know where I stand and what I need to improve” “I can clearly see which module requires more study and improvement” “Satisfying to know where I am excelling” “It is motivational to keep work up to the same standard” “It has made me relax a bit” “It makes me worry less” “I have heightened my expectations of myself as I can see that with a little more work I can push my marks up”

STUDENT RESPONSE

Finalist:BestUseofSumma@veAssessment

Newcastle University Digital Exam Service

3/6/20 LearningandTeachingDevelopmentService 20

Jack Ennis!

Rebecca Gill!

Learning and Teaching Development Service !

Newcastle University!!

Current and planned provision

KeyObjec3ve:Makeonlinesumma3veassessmentpossibleforawiderangeofassessmenttypesandinnon-clusterenvironments

3/6/20 LearningandTeachingDevelopmentService 21

050

100150200

2007-8

2008-9

2009-10

2010-11

2011-12

2012-13

2013-14

2014-15

2015-16

2016-17

2017-18

2018-19

Numberofdigitalexamsdelivered

Academicunits

StudentWellbeing

ExamsOffice LTDS

ITServiceEstates

Transforming Assessment

JISC'TheFutureofAssessment'report(Spring2020)Technologycanbeusedtotransformassessmenttomakeitmore:Authen3cAccessibleAppropriatelyautomatedCon3nuousSecure

JISC,2020hAps://www.jisc.ac.uk/reports/the-future-of-assessment

3/6/20 LearningandTeachingDevelopmentService 22

Evaluation: digital written exams

3/6/20 LearningandTeachingDevelopmentService 23

24%

27%25%

17%

7%

HowdidyourexperienceoftakingthewriPenpartoftheexamoncomputer

comparetoyourpreviousexperiencesoftakingwriPenexamsonpaper?

MuchbeAer

SomewhatbeAer

NeitherbeAernorworse

Somewhatworse

MuchworseStudentevalua3onsofalldigitalwriAenexamstoMarch2020

itwassomucheasiertoeditmywork.Wri3ngisacyclicalprocess,sobeingabletogobackandamendmypreviousparagraphwasreallyhelpful

Evaluation: Bring Your Own Device (BYOD)

LearningandTeachingDevelopmentService 243/6/20

27%

31%

28%

11%3%

HowdidyourexperienceoftakingtheexamonyourlaptopcomparetoyourpreviousexperienceoftakinganonlineexamonaUniversitycomputerina

cluster?

MuchbeAer

SomewhatbeAer

NeitherbeAernorworse

Somewhatworse

MuchworseStudentevalua3onsofallBYODdigitalexamstoMarch2020

BeingabletotypeonakeyboardIamusedtowasgreat

Itwasmucheasiertoconcentrateonlaptopsastherewasnotasmuchofanoisecreatedfrompeopletyping

Evaluation: colleagues’ perspectives

3/6/20 LearningandTeachingDevelopmentService 25

allowedmetoprovideanexamthatiscustomisabletomymoduleinordertomeetmylearningoutcomes[…]Icanassessstudentsonarangeofskillsandknowledge-basedoutcomes. DrAdrianSmall,Opera3onsManagementmoduleleader

Ibelievethattheeaseofreadingeachpaperactuallyledtofairermarking.Studentswithpoorhandwri3ng[…]areomengiventhebenefitofdoubtandthisbiasdisappeared.

DrGabrieleJordanPsychologymoduleleader

allowedmetouseanexternaltool,andIwasabletoaskapplica3onques3onsmuchmorecomplexandinteres3ngasaresult.

DrCharlesMorissetCompu3ngmoduleleader

Opportunities and challenges

•  Examroomtechnicalsupport•  Logis3csatscale•  Assis3vesomwareforlockeddownexams

•  Incorpora3ngdiagramsdrawnonpaper

•  Demandfornewtypesofdigitalexams

•  Covid-19response

3/6/20 LearningandTeachingDevelopmentService 26

Finalist:Innova@on

Anastasios Karakolidis anastasios.karakolidis@gmail.com

Using Animations in Assessment

The reality

Thevastmajorityoftestsinusetodayrelyheavilyonsta3ctext

The problem with text-based assessments Limitedintermsofthecomplexityofwhatcanbepresentedass3muliand/orresponses[1]

Test-takersarerequiredtohaveskills(e.g.,readingcomprehension)thatmaybeirrelevanttotheconstructthattheassessmentpurportstocapture[1,2]

Suchequityandfairnessissuesmightaffect:•  test-takerswithpoorlanguageandreadingskills•  non-na3vespeakers•  peoplewithdyslexia

Construct-irrelevantvarianceàvalidity[3]

This innovation…

explorestheextenttowhichanimatedvideosareausefulalterna3veto

wriAentextintes3ng

Why animated videos?

Advantages:• replacemuchofwriAentext• enablethevisualisa3onofcomplexinforma3onthatcannotbeeasilycommunicatedviatext

• moreeffec3vethanothervideo-basedmediaChallenges:

• developmentcomplexi3es• cost

Development of an animated test Selec3onofasitua3onaljudgementtest(SJT)designedbyStevenStemler,RobertSternbergandtheircolleaguesatYaletomeasureteachers’prac3calknowledgeofhowtodealwithchallengingsocialsitua3onsatwork[4,5]

Collabora3onwithananima3oncompanytocreate15animatedvideosbasedontheoriginaltext-basedscenarios

A sample text-based scenario

The animated SJT

AnimaAonsdevelopedbyAnimatedscenarios.com-PlaEormdesignedbyPsycholate

The animated test

AnimaAonsdevelopedbyAnimatedscenarios.com-PlaEormdesignedbyPsycholate

The animated test

Process of animating 1s

t dram

2nd d

ram

Finaldram

The research study Experimentalcomparisonoftheanimatedvs.thetext-basedversionofthetest

Sample:na3ve(n=51)andnon-na3ve(n=78)English-speakingpre-serviceteachers-randomassignment

Measures:• SJTperformance• Englishproficiency• readingcomprehensionabilityinEnglish

• ThevarianceaAributedtoconstruct-irrelevantfactorswasreducedby9.5%

• Inotherwords,theuseofanimatedvideosreducedthedependencyoftestperformanceonirrelevantfactors,suchaspar3cipants’proficiencyandreadingcomprehensioninEnglish

Research question #1: What impact does the use of animated videos have on construct-irrelevant variance attributed to language and reading skills?

Research question #2: What impact does the use of animated videos have on test-takers’ reactions to the test?

Theanimatedversionofthetest

wasperceivedasmorevalidandfair(d=0.38)

wasperceivedasmoreenjoyable(d=0.50)

wasfoundtouselanguagethatwaslessdifficulttounderstand(η2=0.043)

So what? • Didanima3onsmakeasubstan3aldifference?• Weshouldcareaboutwhattest-takersthinkoftheteststheytake!

• Test-takerspercep3onsofatestcanhaveanimpacton:•  theirastudestowardsthebodyorganisingtheassessment[1]•  theirengagementwiththeassessment[6,7]•  theirperformance[6,7]

• Isitworththeeffortandmoney?

The potential • Improvingequityandfairness

• Improvingstandardisa3oninassessment

• Tes3ngacrossdifferentpopula3ons

• Tes3nghard-to-measureconstructs

• Professionaldevelopment

References [1]E.C.Popp,K.Tuzinski,andM.Fetzer,“Actororavatar?Considera3onsinselec3ngappropriateformatsforassessmentcontent,”in

TechnologyandTes3ng:ImprovingEduca3onalandPsychologicalMeasurement,F.Drasgow,Ed.NewYork,NY:Routledge,2016,pp.79–103.

[2]J.Abedi,“Linguis3cfactorsintheassessmentofEnglishlanguagelearners,”inTheSAGEHandbookofMeasurement,G.Walford,E.Tucker,andM.Viswanathan,Eds.California,Ca:SAGE,2010,pp.129–105.

[3]AmericanEduca3onalResearchAssocia3on(AERA),AmericanPsychologicalAssocia3on(APA),andNa3onalCouncilonMeasurementinEduca3on(NCME),StandardsforEduca3onalandPsychologicalTes3ng.Washington,DC:AmericanEduca3onalResearchAssocia3on,2014.

[4]S.E.Stemler,J.G.EllioA,E.L.Grigorenko,andR.J.Sternberg,“There’smoretoteachingthaninstruc3on:Sevenstrategiesfordealingwiththeprac3calsideofteaching,”Educ.Stud.,vol.32,no.1,pp.101–118,2006.

[5]J.G.EllioA,S.E.Stemler,R.J.Sternberg,E.L.Grigorenko,andN.Hoffman,“Thesociallyskilledteacherandthedevelopmentoftacitknowledge,”Br.Educ.Res.J.,vol.37,no.1,pp.83–103,Feb.2011.

[6]Chan,D.,SchmiA,N.,DeShon,R.P.,Clause,C.S.,&Delbridge,K.(1997).Reac3onstocogni3veabilitytests:Therela3onshipsbetweenrace,testperformance,facevaliditypercep3ons,andtest-takingmo3va3on.JournalofAppliedPsychology,82(2),300–310.

[7]ScoA,J.C.,&Mead,A.D.(2011).Founda3onsformeasurement.InN.Tippins&S.Adler(Eds.),Technolgy-EnhancedAssessmentofTalent(pp.21–65).SanFrancisco,CA:Jossey-Bass.

Emailaddress:anastasios.karakolidis@gmail.com

e-AssessmentAssocia@on

JointheeAAforfreeatwww.e-assessment.com/registerJointheeAALinkedIngroupandaddtothediscussionsFollowusontwiAer@eAssessSpreadthewordaboutthe2021Awardsprogramme.Findoutmoreatwww.eassessmentawards.comGetintouch!Sendinyourblogposts,thoughts,queries,casestudiesandnewstomarke3ng@e-assessment.com

Thankyouforlistening

Webinar Session feedback With thanks from your hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com Recording available http://transformingassessment.com

Next session 15 July 2020 Assessment in

Higher Education Conference

Preview for 2021 Register

http://taw.fi /15jul2020

Webinar Series

e-Assessment SIG

@transformassess