openresearchonlineoro.open.ac.uk/70079/2/alrashidi et al. - 2020 - a...alrashidi, huda; ullmann,...

Open Research OnlineThe Open University’s repository of research publicationsand other research outputs

A Framework for Assessing Reflective Writing ProducedWithin the Context of Computer Science EducationConference or Workshop ItemHow to cite:

Alrashidi, Huda; Ullmann, Thomas; Ghounaim, Samiah and Joy, Mike (2020). A Framework for AssessingReflective Writing Produced Within the Context of Computer Science Education. In: Companion Proceedings 10thInternational Conference on Learning Analytics & Knowledge (LAK20, 24/03/2020, Frankfurt, Germany.

For guidance on citations see FAQs.

c© [not recorded]

Version: Accepted Manuscript

Link(s) to article on publisher’s website:https://lak20.solaresearch.org/

Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.

oro.open.ac.uk

http://oro.open.ac.uk/help/helpfaq.html

http://oro.open.ac.uk/help/helpfaq.html#Unrecorded_information_on_coversheet

https://lak20.solaresearch.org/

http://oro.open.ac.uk/policies.html

CompanionProceedings10thInternationalConferenceonLearningAnalytics&Knowledge(LAK20)

CreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0)

1

A Framework for Assessing Reflective Writing Produced Within the Context of Computer Science Education

HudaAlrashidi1,4,ThomasDanielUllmann2,SamiahGhounaim3,MikeJoy1

ComputerScienceDepartment,UniversityofWarwick1InstitutionInstituteofEducationalTechnology,TheOpenUniversity2

CentreforAppliedLinguistics,[email protected]

ABSTRACT: Reflective writing is known to be an effective activity to increase students'learning.However, there is limited literature inreflectivewritingassessmentcriteria in thecontextofcomputerscience (CS)education. In thispaper,weaimtoexploreameaningfulreflectivewritingassessmentcharacteristics.ThathasbeenusedtoassessreflectivetextbyCS educators. This paper has two contributions: (a) we developed a Reflective WritingFramework(RWF)forthemaincriteriahasbeenusedtoassessreflectivetextinCSeducationfromthefindingsofasemi-structurequestionnaire;(b)theRWFwastestedempiricallyusingapilottestofthemanualannotationusedtomodifytheframework.Thisanalysisresultedinaninter-raterreliabilityof0.78beingachieved.Theoverallgoalofthisresearchistodevelopa Learning Analytics (LA) tool which can automatically detect the categories of the RWFpresentinatexttoassessthestudentauthors’reflectivewritinginrelationtoCS.

Keywords:ReflectiveWriting,ComputerScience,Reflection,ReflectionDetection,ReflectiveWritingAnalytics,LearningAnalytics

1 INTRODUCTION

LearningAnalytics(LA)isgraduallybecomingoneofthepivotalaspectsofeducationaltechnology.ThispaperinvestigatesanLAtoolthatsupportsreflectionbyanalyzingandprovidingfeedbackonreflectivewriting(RW).RWcansupportstudentstogainawarenessoftheirlearningprocesses.Interms of Computer Science (CS), “reflection is worth encouraging, for its indirect effect on thetechnical skills and knowledge which are our ultimate purpose in teaching computer science”(Fekete,Kay,Kingston,&Wimalaratne,2000).TechnicalskillsareatthecoreofCS,andthesecenteraroundformulatingproblemsandtheirsolutions.Sincereflectionisametacognitiveprocess,itcanonlybeassessedindirectly-throughwrittenorverbalforms.AnalyzingRWmanuallymakesgivingstudentsfeedbackachallengingandtime-consumingtask.Automatedfeedbackcanbettersupportthestudentsintermsofprovidingtimelyanalyses.LAtoolshavethegoalofsupportingreflection–specifically,byanalyzingstudents'reflectivetexts.TodesignanLAtoolforRW,thereisanecessityeithertoadaptanexistingmethodologyortodevelopanewframeworkforthispurpose(Gibsonetal.,2017).ThisstudyaimstodevelopaRWFramework(RWF)forCSeducationtodevelopanLAtoolforRW.Wefocusonthefollowingresearchquestions:1)whatarethecharacteristicsofRWwithinCSeducation?And2)whataretheindicatorswhichcanbeusedtoassessRWlevelsastheyoccurinCSeducation?



2

2 RELATEDWORK

ReflectiveactivitiesthathavebeenusedrecentlyinvestigatedRWinCSeducation(Alrashidi,Joy,&Ullmann,2019;George,2002;Stone&Madigan,2007)astheyhaveinotherdisciplinessuchassocialandhealthsciences.However,theliteratureonRWinCSeducationislimited.Forinstance,George(2002)andFeketeetal. (2000) investigatedusingthereflective journal intermsofbenefits tothestudentsinanundergraduateprogrammingcourse.Bothstudiesnotedthatreflectivejournalswerebeneficial toget students to reflecton their softwaredevelopmentprocessesas it ispartof theirlearningoutcome.Moreover,inaccordancewiththeLAtoolforreflectioninCSeducation,Dorodchietal.(2018)implementedanactivitybasedontheCScoursewithperiodicreflectionbyapplyingKolb’slearningmodel.TheyvalidatedtheresultofstudentreflectionthroughtheLAclassificationmodel.Itconcluded that including reflection as a feature could improve the accuracy and time of theirclassificationmodel. However, there isadifficulty forstudents to reflecteffectivelyon theirownunderstanding.MoskalandWass(2019)developedanapproachforeducatorstoencouragestudentstothinkabouttheirsoftwaredevelopmentstepsthroughaseriesofsessions.Theyfoundthattheapproachwasbeneficialforbothstudentsandeducators.However,Grossman(2009)mentionedthatanumberofstudentsdidnotunderstandwhattheyareexpectedtoreflectonduetolackofguidance.Grossman’s(2009)findingsprovidereasoningforthestudyconductedbyGeorge(2002)thatfoundreflective journalasnotwidelyacceptedbystudentsand/oreducators inCSeducation. TheRWFdeveloped here is a guideline for students to determine themain elements on which they areexpectedtoreflect,andforeducatorsonassessingtheirstudents’RW.

3 THERWF

Semi-structuredquestionnairesexploredperspectivesof6HEexperts(Exp.)–selectedbasedontheirbreadthofacademicskillsinCSandtheirknowledgeofreflection–onRWlevels,andtheindicatorstheyusetoassessRWinCSeduaction.Athematicanalysisoftheresponsesresultedinthreecodesforlevelsofreflection:1)non-reflective,2)reflective,and3)criticallyreflective;andsevencodesforindicatorssummarizedasfollows.

First,thedescriptive:twoexpertsusedsimilarwords in theirdefinitionsofsuch indicators.Exp.Astatedthat:“studentsmerelydescribewhattheyhavedone…withoutanyexamples.”Exp.Cusedtheword “listing” instead stating that “Iwouldoften see simple summariesof lesson content, orlistingsoftopicscoveredthatIwouldclassasnon-reflective”.Thismeansthat“non-reflective”textsaresuperficialdescriptionsofsituations.Second,theunderstanding:alltheexpertscharacterizeditasborderingon the reflective level. Forexample,Exp.Edefined this indicatoras, “whenstudentsidentify their understanding of competencies … [RW] has been reached.” Accordingly, theunderstanding indicator characterizes both the non-reflective and the reflective levels, per thecontext.Third,thefeeling:theexpertsarguedthatthereflectivelevelapplieswhenthewritercanidentifytheirownthoughtsandfeelings.Forexample,Exp.Cstatedthat“Iwouldlookforevidenceofwhatthestudentspreviouslythoughtorfeltonwhetherthathadworkedornot.”Thismeansthatthefeelingindicatorintheproposedframeworkcanbeeitheratthereflectiveorcriticallyreflectivelevel.Fourth,reasoning:theyargueditoccurswhenawriterexplainsasituation/issuebyprovidingexamples/causes. For instance, studentswould “clearlyexplain their process,whatworked,whatdidn’t”(Exp.D),and“provideexamples”(Exp.G),and/or“analysisofproblemsand[theirsolutions]”(Exp. C). Fifth, perspective: this could be detected when “Students share personal thoughts and



3

connectwithotherthoughts”(Exp.G),andgiving“evidenceofre-evaluation[dueto]feedbackfromothers” (Exp. D). Both experts emphasized that perspective takes into consideration others’perspectives.Exp.D summarizes itas students’ ability “to connect the topic in question towiderapplicationsinthediscipline,theircommunity,ortheworld”.

Sixth,thesignificanceofthenewlearningindicatorwasclearlyemphasizedbythepanels.Theexpertscommentedthattheysearchforevidenceoflearning.Forexample,Exp.Hmentionedthatthestudentmust show evidence of what has been learnt in terms of personal and professional skills by“connecting what we have learned and the skills… gained to our own personal or professionaldevelopments”.Lastly,futureaction:thepanelofexpertscommentedthattheysearchforevidenceofoutcomewhenassessingRW.Exp.Cexpectedthestudenttoshowtheyhadachievedadeeperunderstandingoftheproblemtheywereengagedwith,asaresultofproducingtheRW,intermsofcognitionbyhaving “adeeperunderstanding ofwhat they have learnt”,metacognitionbybeing“betterabletomanagetheirownlearninganddevelopmentoncetheyleaveformaleducation,”andsociallywiththeability“toworkbetterinateambyidentifyingandowningtheirownweaknesses,andsharingtheirsuccesses.”

Table1showsalltheindicatorsandlevelsofourRWFwhichisconsistentwiththeliteratureonRWandonreflectiontheories,especiallyintermsofthelevelsdefinedbyWong,Kember,Chung,andYan(1995)andthereflectionindicatorsdefinedbyUllmann(2019).Table1LevelsandIndicatorsoftheRWFforCS

Reflectivelevels Indicators

Non-Reflective Descriptive:thewriterreportsafactfromexperienceand/ormaterials

Understanding:thewriterunderstandsand/oranalysestheexperience.

Reflective Feelings:thewriteridentifiesand/oranalysestheirownthoughtsandfeelings.

Reasoning:thewriterexplainstheexperiencebygivingreasons.

Critically Perspective:thewritershowsawarenessofalternatives.

Reflective Newlearning:thewriterintegratesand/ordescribesnewlearning

Futureaction:thewriterintendsand/orplanstodosomethinginthefuture.

4 VALIDATIONOFTHERWF

Amanualannotationaimedtoproduceafinalversionoftheframeworkthroughmanualreviewsandusingthisactivityasabasisforaniterativecycleofframeworkdevelopment.Thedatasetconsistedof30RWdocuments–split into360sentences–from30computersciencestudents inmoduleCS310ComputerScienceProject all relating toa3rd-yearprojectundertakenduring2013–2016academicyears.ThedatawerecollectedbytheCSDepartmentattheauthors’universityaspartofitsnormalassessmentprocessandthenprovidedtotheresearchersfullyanonymized.FourpilotstudieswereconductedOctober2018–May2019toproducereliableguidelinesbasedontheRWFanddevelopedviatheraters’commentsandsuggestions.InTable2,thefirstpilotstudy,fourindependentratersappliedtheinitialRWFtotheannotationof20 sentences and then explained their ratings. From this, we recognized some ambiguity in thereflectionindicatorsasformulatedintheguidelinesgiventotheraters.Inthesecondpilotstudy,thethree independent raters applied themodifiedRWF to 40 random sentences. ThemodifiedRWFenabledthemtoreachaconsensusregardingthethreelevelsandthesevenindicators.Someminor



4

areasoftheRWFguidelineswerethenrefined.Inthethirdandfourthpilotstudies,twoindependentratersappliedtheRWFasframedafter improvements.Akappastatistic (k)usedtodeterminetheinter-raterreliabilityandadjustforthepossibilityofachanceagreementbetweenthecoders.Theinter-raterreliabilityof0.87and0.78,respectively,whichwassubstantialtoalmostperfectagreement(Landis&Koch,1977).Table2:Theinter-raterreliabilitycomputedforeachiterationoftheRWFduringthefourpilottests

Dateofthepilottest #iteration Sample #raters kOctober2018 1 20 4 0.52January2018 2 40 3 0.73March2019 3 100 2 0.87May2019 4 200 2 0.78

5 CONCLUSIONANDFUTUREWORK

ThisresearchhasansweredtworesearchquestionsthatexploredthecharacteristicsofRWtoidentifythe assessment indicators and the levels relating toRW in CS education. Based on the thematicanalysis of the questionnaire, the RW framework was proposed; this has three levels and sevenindicatorstoassessRWproducedinthecontextofCSeducation.ThefutureworkwillbeusingthefindingstoproducealabeleddatasettouseittodevelopanLAtool.ThatwillautomateRWanalysisbasedonmachinelearningandrule-basedapproachesfordeterminingthefeaturesofRW.

ACKNOWLEDGEMENT

TheresearchwaspartialyfundedbyKuwaitFoundationfortheAdvancementofSciences(KFAS);ProjectcodeCB19-68SM-01;andKindSaudbinAbdulazizUniversityforHealthSciences(KSAU-HS);scholarshipprogramforTeacherDevelopment,DepartmentofLanguagesandCultures.

REFERENCES

Alrashidi,H.,Joy,M.,&Ullmann,T.(2019).AReflectiveWritingFrameworkforComputingEducation.Paper presented at the ITiCSE 2019: Proceedings of the 24th Annual ACMConference onInnovationandTechnologyinComputerScienceEducation,Aberdeen,UK.

Dorodchi,M.,Benedict,A.,Desai,D.,Mahzoon,M.J.,MacNeil,S.,&Dehbozorgi,N.(2018).Designand Implementationof anActivity-Based IntroductoryComputer ScienceCourse (CS1)withPeriodic Reflections Validated by Learning Analytics. Paper presented at the 2018 IEEEFrontiersinEducationConference(FIE).

Fekete, A., Kay, J., Kingston, J., & Wimalaratne, K. (2000). Supporting reflection in introductorycomputerscience.PaperpresentedattheACMSIGCSEBulletin.

George,S.E. (2002). Learningand the reflective journal incomputerscience.AustralianComputerScienceCommunications,24(1),77-86.

Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., & Knight, S. (2017).Reflectivewritinganalytics foractionable feedback.Paperpresentedat theProceedingsoftheSeventhInternationalLearningAnalytics&KnowledgeConferenceon-LAK'17.

Grossman,R.(2009).Structuresforfacilitatingstudentreflection.CollegeTeaching,57(1),15-22.Landis, J. R.,&Koch,G.G. (1977). Themeasurementof observer agreement for categorical data.

biometrics,159-174.



5

Moskal,A.C.M.,&Wass,R. (2019). Interpersonalprocessrecall:anovelapproachto illuminatingstudents’ software development processes. Computer Science Education, 29(1), 5-22.doi:10.1080/08993408.2018.1542190

Stone,J.A.,&Madigan,E.M.(2007).IntegratingreflectivewritinginCS/IS.ACMSIGCSEBulletin,39(2),42-45.

Ullmann, T. D. (2019). Automated Analysis of Reflection in Writing: Validating Machine LearningApproaches.InternationalJournalofArtificialIntelligenceinEducation,1-41.

Wong,F.K.,Kember,D.,Chung,L.Y.,&Yan,L.(1995).Assessingthelevelofstudentreflectionfromreflectivejournals.Journalofadvancednursing,22(1),48-57.

openresearchonlineoro.open.ac.uk/70079/2/alrashidi et al. - 2020 - a...alrashidi, huda; ullmann,...

Documents