towards the discovery of learner metacognition from reflective … · reflective writing, an...
TRANSCRIPT
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 22
Towards the Discovery of Learner Metacognition from Reflective Writing
AndrewGibsonQueenslandUniversityofTechnology,AustraliaUniversityofTechnologySydney,Australia
KirstyKittoQueenslandUniversityofTechnology,Australia
PeterBruzaQueenslandUniversityofTechnology,Australia
ABSTRACT:Modernsocietydemandsrenewedattentiononthecompetencies requiredtobestequipstudentsforadynamicanduncertainfuture.Wepresentexploratoryworkbasedonthepremisethatmetacognitiveandreflectivecompetenciesareessentialforthistask.Bringingtheconcepts of metacognition and reflection together into a conceptual model within which weconceived of them as both a set of similar features, and as a spectrum ranging from theunconsciousinner-selfthroughtotheconscious,external,socialself.Thismodelwasusedtoguideexploratory computational analysis of 6,090 instances of reflective writing authored byundergraduatestudents.Wefoundtheconceptualmodelusefulininformingthecomputationalanalysis,whichinturnshowedpotentialforautomatingthediscoveryofmetacognitiveactivityinreflectivewriting,anapproachthatholdspromiseforthegenerationofformativefeedbackforstudentsastheyworktowardsdevelopingcore21stcenturycompetencies.
Keywords:Metacognition,reflection,reflectivewritinganalytics,computationalanalysis,naturallanguageprocessing,learninganalytics,21stcenturycompetencies
1 INTRODUCTION
Continuousrapidchangeisadominantfeatureofmodernsociety.Technologyisrapidlyevolving,andthisisleadingtothedemiseofentireemploymentsectors,andthecreationofcompletelynewones.Withinthiscontext,wecanhardlyanticipatewhatskills,knowledge,andexperiencethestudentsofourformaleducation systems are going to need to know upon graduation, let alone ten years into the future.Learninghowtoacquirenewideas,skillsandapproachesforthemselvesisanimportantstrategythatwillhelpourstudentstosucceedinthisageofuncertainty.Learningtolearn,andtherelatedconceptsofself-directedlearning,self-regulatedlearning,andindependentlearning,arecertainlynotrecentpedagogicalideas(Herber&NelsonHerber,1987;Paul,1990),butthegeneralapproachhas,overtime,gainedgreaterprominence(Cornford,2002;Takanishi,2015;Wilson,2015)aswehavewitnessedsignificantchangesin
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 23
variousdomainsofknowledge(CEDA,2015).Whilechangesinthebodyofknowledgehavealwaysbeenwithus, itseemsthattheyhaveneverbeenasrapidastheyarenow,andthiscurrentaccelerationofchangeappearstobe increasingratherthanabating.Wecannotescapethiscontext,andwithin itwemust aim to educate our students in away that prepares them for the future, equipping themwithadaptivecapabilitiesthatcanflourishinthisfluidenvironment.
Historically,formaleducationhasfocusedonknowledgeoffacts,andwhilefoundationalcontentisstillseen as critical, entire domains are starting to recognize the need to shift educational focus frompresentinglargeamountsofcontent,towardsinstillingakindofmeta-knowledge;thatis,thecapabilityforunderstandingwhatknowledgemaybeneededforasituationandwhentoapply it,aswellastheabilitytofindtheidentifiedrequisiteknowledge.Developmentofautonomyinthisrespectisnotpossiblefor the learnerwithout themdrawinguponmetacognitiveabilities toevaluate their current cognitiveposition, and formulate strategies for advancing it. Despite the changes in formal education toaccommodatethisparadigmshift,furtherworkisrequired,andintelligenttoolsareneededtosupportit.
Thismoderneducationalcontexthasprovidedanunderlyingmotivationfortheworkpresentedhere.Inthispaper,wedemonstratepotentialforcomputationalanalysisofreflectivewriting(ReflectiveWritingAnalytics) as a means of discovering evidence of metacognitive activity in the reflective writing of alearner.Asweshall see inSection2, thedistinctionbetweenmetacognitionand reflection isblurred.However,theliteraturesuggestssignificantrelationshipsbetweenthem,andthattheanalysisofwrittenreflectionmayallowforthediscoveryofmetacognitiveactivityonthepartofthelearner.Anempiricalstudywasdesignedtoexplorethisintuition.
Theresearchwasconductedintwoparts,thefirstbeingthedevelopmentofaconceptualmodeldrawnfromthemetacognitionandreflectionliterature(Section2),andthesecondbeinganexplorationofthemodel’sapplicationtothecomputationalanalysisofReflectiveWritingAnalytics(Section3).Weconcludewithanexaminationoftheimplicationsofthisstudytothefieldoflearninganalytics(Section4).
2 METACOGNITION AND REFLECTION
InthecenturysinceJohnDewey introducedtheconceptofreflection ineducation(Dewey,1916),theeducational community has increasingly considered reflection an important part of learning. This hasbecomemorefocusedsinceFlavell(1976)definedthetermmetacognition,whichencouragededucatorsto embrace the significance of thinking about thinking. In a pedagogical context, both reflection andmetacognition regularly co-occur, suggesting a strong link between them (Schön, 1983;Moon, 1999).IndeedBoud,Keogh,andWalker(1985,p.141)statethat“Reflectionisthus‘meta-thinking’”andSanders(2009,p.688)assertsthat“reflectionisametacognitiveprocess.”Thislinkholdssignificanceforourwork,sincewhilemetacognitioncanbehiddenfromview,reflectioncanbemadeexplicitinstudents’reflectivewriting.Ourintuitionisthatexploitingthislinkbetweenreflectionandmetacognitionwillprovideameansfordiscoveringevidenceofmetacognitioninthereflectivewritingofthelearner.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 24
Inthissection,weexaminesomeofthekeyliteratureonmetacognitionandreflection,consideringeachin turn as individual concepts. We then identify key similarities and differences between them, andproposeaconceptualmodelthatencapsulatesthesefeatures.
2.1 Metacognition
John Flavell originally defined the term metacognition as “the active monitoring and consequentregulationandorchestration”of informationprocessingactivities,“usually inserviceofsomeconcretegoalorobjective”(1976,p.232).Flavell’s(1979)definitionmaturedintoamodelofcognitivemonitoringthatcomprisedmetacognitiveknowledge,metacognitiveexperiences,goals,andactions.Henotedthatthesefourcomponentscontinuallyinteractthroughoutthecognitiveprocess.Thesecomponentsprovideanindicationofhowmetacognitionmaypresentifitweremadevisibleinstudentwriting.
StudiesconductedsubsequenttoFlavell’sworkhaveidentifiedtwotypesofmetacognition:regulatorystrategies,andstrategicknowledge (Quirk,2006).Regulatory strategiesalignwith themonitoringandregulationaspectsofFlavell’sdefinition,whereasstrategicknowledgeincorporatestheorchestrationandgoalaspects.Thisapparentlydualnatureofmetacognition isalso identifiedbyothers,butnotalwaysalong thesame lines.Forexample,Amseletal. (2008)examined the relationshipofmetacognition toscientificreasoning,explicitlytakingadualprocessperspective:“Indual-processtheory,metacognitiveskillsfunctiontoregulateconflictsbetweenanalyticallyandexperientiallybasedresponses”(Amseletal.,2008, p. 454). Amsel et al. defines the experiential as heuristic and automatic, comparing it to theanalytical,whichisformalandsystematic.ThesedimensionsappeartoaccordsomewhatwithFlavell’s1979modelwithmetacognitionexperienceandknowledge;however,itseemslikelythatFlavellwouldnot takeAmseletal.’sviewofmetacognitionasamoderatingskill thatoverrides theexperiential, forAmseletal.’sdefinitionisspecifictoastyleofcognitionthatpresentsasscientificreasoning,andmaynottranslatetootherformsofcognition.Lehmann,Hähnlein,andIfenthaler(2014)alsopromotedadualviewof metacognition within a widermodel of self-regulated learning. They note thatmetacognition caninclude a structural component that is relatively stable, including task knowledge and knowledge ofcognition,andaprocesscomponentthat ismore immediatetoaparticularsituation.Thiscomponentincludesplanning,monitoring,andevaluation.WenotethesimilaritiesbetweenthesecomponentsandFlavell’sgoals,actions,andregulatorystrategies.
The view that metacognition is a trainable skill, and that people should be encouraged to be moremetacognitive,hasexisted for as longas the term itself. IndeedFlavell (1979) asserted that cognitivemonitoring is trainable,and that it shouldbepromotedmorewidely: “Iamabsolutelyconvinced thatthereis,overall,fartoolittleratherthanenoughortoomuchcognitivemonitoringinthisworld”(p.910).However,weshouldbecarefultoavoidtheconclusionthatmoreisbetter,asexcessivemetacognitionisevident inpeoplewhoarepathologicallyobsessive,andwhennon-selective, cognitivemonitoringcanbecome rumination on negative outcomes, leading to anxiety related problems (Quirk, 2006). Thissuggeststhatdiscoveryofmetacognitionshouldbemoreconcernedaboutitsqualitiesratherthanthequantityinwhichitoccurs.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 25
Metacognitionhasoftenbeenassociatedwithintentionalandconsciouslyaccessiblethought;however,thisdoesnotrepresentacompletepicture,andignorescognitiveactivitythatoccurswithoutawareness.“Indeed,theconsciouscharacterofmetacognitionwaschallengedbyearlyresearchonmeta-memoryshowingthatpeoplemonitorandcontroltheircognitionwithoutbeingconsciouslyawarethattheydoso”(Efklides,2008,p.281).Koriat(2000)suggestedthatmetacognitionactuallypossessestwofaces:animplicit automatic mode and an explicit controlled mode. In fact, he proposes that metacognition,particularlymetacognitiveexperience,mediatesbetweentwolayersofconsciousnessstatingthat“theyserve to interface between implicit-unconscious-automatic processes on the one hand, and explicit-conscious-controlledprocessesontheother”(p.152).Whileeducationistypicallyfocusedontheexplicitcontrolledmodeofmetacognition, it istheexistenceoftheimplicitautomaticmodeofmetacognitionthatwesuggestdifferentiatesitfromtheactivityofreflection.
2.2 Reflection
Somewhat likemetacognition, reflection tends to be a concept recalcitrant to crisp definition.Moon(1999)notedthatasingledefinitionofreflectioniselusive,despiteattentiontothetopicovermanyyears.She observed that “reflection seems to be a form of mental processing with a purpose and/or ananticipatedoutcomethatisappliedtorelativelycomplicatedorunstructuredideasforwhichthereisnotanobvioussolution”(Moon,1999,p.98).Thismentalprocessingaspectappearstobesuchacommonhumanactivitythatweareindangerofmissingtheimportanceoftheconceptaltogether.Indeed,Boudetal.(1985)identifiedfamiliarityasacontributortoourlackofunderstandingaboutreflection:
Theactivityofreflectionissofamiliarthat,asteachersortrainers,weoftenoverlookitinformallearning settings, andmake assumptions about the fact that not only is it occurring, but it isoccurringeffectivelyforeveryoneinthegroup.Itiseasytoneglectasitissomethingwhichwecannotdirectlyobserveandwhichisuniquetoeachlearner.(p.8)
Reflection can also be understood differently in different contexts. In this paper, we are particularlyconcernedwithreflectionforlearningwithinaneducationalcontext.Howeveritisworthkeepinginmindthatthetermisalsocommonlyusedwithrespecttolearningwithinaprofessionalcontext(i.e.,reflectivepractice),andinapersonalsense(i.e.,personalreflection).Thesedifferentapplicationsofreflectionarenotnecessarilyindependentfromeachother,andsharemanyofthesamefeatures.Asaresult,researchdirectedatoneapplication isoftenreferredto inthecontextofanother.Forexample,Schön’s(1983)work,whichfocusedonprofessionalreflectivepractice,hasoftenbeenusedasabasisforunderstandingreflectionwithinaformaleducationcontext.
Furthercomplicatingtheconcept,reflectionasamentalactivitycanbeeasilyconflatedwithreflectivewriting—a task designed to elicitmental reflective activity, or in the case of assessment, documentmentalreflectiveactivity.Forthepurposesofthisstudy,weconsideredreflectiontobeatypeofmentalactivitythatcanbeevidencedbyreflectivewriting.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 26
Despitetheambiguityaroundtheterm,Boudetal.(1985)identifiedthreefeaturesofreflectionthattheyclearlyobserved.First,asreflectionisapersonalactivity,thelearneriscentralandincontrol,revealingonlywhat theywish aboutwhat they have reflected upon. Second, reflection is goal orientated andpurposive.Third, reflection isacomplexprocesswithanaffectivedimensionthat interactswithothercognitiveprocesses.Thesethreeaspectsofreflectioncorrelatewellwiththeregulation,planning,andexperientialqualitiesidentifiedinmetacognition.
However,unliketheliteratureonmetacognition,thereisagrowingbodyofliteraturethathighlightstheimportanceofasocialdimensioninreflection.Indeedsomewouldarguethatreflectioncannotbefullyunderstoodwithoutexamining itwithina social context.Ryan (2011)proposesamulti-levelmodel—stronglylinkedtothecontextoftheauthor—forteachingandassessingacademicreflection.Thisworkviewsreflectionasasociallysituatedactivityinvolvingadeepexaminationofselfinrelationtothesocialcontext,andtheintentiontotakeactiononthebasisofthatexamination.
ActionisalsoakeyaspectofMezirow’s(1990)workontransformativelearning.Mezirow(1990)proposeda high level of reflection called critical reflection,which involves learners critiquing their pre-existingknowledgebasedonnewinformation,holding“biasesinabeyance,and,throughacriticalreviewoftheevidenceandarguments,make[ing]adeterminationaboutthejustifiabilityoftheexpressedideawhosemeaningiscontested”(p.10).Thisworkcertainlyhighlightedtheimportanceofreflectioninlearning,apointthatRyan(2013)extendedbeyondimprovingone’sownunderstandingtotheverysustenanceofthelearningprocess.Sheargued“thatstudentscanandshouldbetaughthowtoreflectindeep,criticalandtransformativewaystoengendersustainablelearningpractices”(p.145).Hereweseesomesimilaritybetweentheapplicationofreflectionto learningandFlavell’sassertionthatmetacognitivemonitoringandcontrolistrainable.
AlsorelatingreflectionandlearningistheworkofZimmerman(2002)whoappliedreflectionspecificallyto the process of creating the self-regulated learner. He stated that “these learners monitor theirbehaviourintermsoftheirgoalsandself-reflectontheirincreasingeffectiveness”(p.66).Hereweseein“monitoring” and “goals” that again there is alignment between conceptions of reflection andmetacognitionfurthersupportingourintuitionthatthetwoarestronglyrelated.
Zimmerman(2002)conceptualizedself-reflectionintermsofself-judgmentandself-reaction,withself-judgment involvingthecomparisonofthe learners’ownunderstandingoftheirperformanceagainstastandard;andself-reactioninvolvinganaffectiveelement(e.g.,self-satisfaction)andanadaptiveelement(e.g.,adjustmentofstrategy).Thistoodisplayssimilaritieswithmetacognition,withself-judgmentandself-reactionrelatingnicelytothejudgmentandaffectivedimensionsofmetacognitiveexperience.
In relating the various understandings of reflection evident in the above literature, we can see thatreflectionisusuallyconceivedofasadeliberateprocesswithatriggerofsomesortandanunderlyinggoalorobjective.Ittendstobepersonalinnature,engagingfeelingsandemotions;however,itisalsosociallysituatedandthereforenotsolelyaninternalactivity.Italsotendstoincludealevelofmonitoringofthe
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 27
particularsituation inquestion,andregulationofbehaviourtowardsachievementof thedesiredgoal.Thereisasensethattruereflectionhasdepthandisadifficultprocess,butonethatcanbetransformative.
2.3 A Model for Metacognition and Reflection
Wehavealreadyidentifiedsomesignificantsimilaritiesbetweenfeaturesofmetacognitionandreflection.Thesesimilarfeaturesprovideafoundationforamodel,butwemustremainconsciousofthedifferencesbetweenthesetwoconcepts.Themostnotabledifferenceisthelackofanimplicitautomatedversionofreflection,andtheabsenceofasociallysituatedtypeofmetacognition.Wefoundsomeresolutiontothese differences in Efklides’ (2008) multilevel model of metacognition. She proposed a model thatcomprisesthreelevels:anon-consciousobjectlevel,themetalevelassociatedwithpersonalawareness,andthemeta-metalevelassociatedwithsocialawareness.Significantly,thisapproachnotonlyrelatesthe non-conscious and conscious aspects ofmetacognition, but also introduces the idea of reflectiontogetherwithmetacognitionbetweenthepersonalawarenessandsociallevels.
Figure1:Aspectrumviewofmetacognitionandreflection.
We conceptualized this difference in the relationship between metacognition and reflection as aspectrum,withtheinternalinner-selfononeendandtheexternalsocial-selfontheother(Figure1).Formetacognition, the leftsideof thespectrumrepresents the implicit,automated,non-consciousmode,whilethecentreincludestheexplicit,conscious,controlledmode(Koriat,2000).Forreflection,therightsideof thespectrumrepresents theexternal, socially situateddimension,and thecentre includes thepersonal,internal,butconsciousaspect.Wesuggestthatthisunderstandingoftherelationshipbetweenmetacognition and reflection not only caters to both implicit–internal and explicit–externalunderstandingsoftheseactivities,butalsoprovidesawayofrelatingtheexplicitmetacognitiontointernalreflectioninthecentreofthespectrum.Inthisarea,thetermscouldbeusedinterchangeably,whileontheleftandrightextremesthetermshavequitedifferentmeanings.Weproposethismodelasawayofembracingthevaryingdefinitionsofmetacognitionandreflectionfoundintheliteraturewhileprovidingadditionalclarityontheuseofeachterm.
However,thespectrumviewprovidedinFigure1doesnotclearlyspecifythewaysinwhichmetacognitionand reflection are alike. To address this, we modelled similar features as interrelated components
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 28
(Figure2).Reducingawholeideaintocomponentpartscanbeproblematic,particularlywhentryingtounderstandhowthecomponentsrelatetothemoregeneralspectrummodel.Thecommonfeaturesofmetacognition and reflection do not explain the changing nature of the two activities as viewed ondifferentpartsoftheproposedspectruminFigure1,andwhilethespectrumhighlightsthegeneralnatureofbothactivities,itdoesnotidentifythecommonfeatures.Soourtwoviewsoftherelationshipbetweenmetacognitionandreflectionneedtobetakentogetherandunderstoodastwoperspectivesoftheonephenomena.Selectingasuitablelevelofgranularityforthecomponentmodelwasimportant.Toofewcomponents resulted in themodel lacking explanatory power, and toomany components resulted insomefeaturesappearingfalselysignificant.
Figure2:Acommonfeatureviewofmetacognitionandreflection.
Ourfinalcomponentmodel(Figure2) istheresultofanumberof iterations,withthemostsignificantchangebeingtheintegrationofthetriggerandgoalcomponentsintothemaincomponentofregulationtogetherwithmonitor and control.Weattempted to address the granularity issueby retaining theseelementsassubcomponentstobemappedintheanalysis,butconsideredthemtogetherwithmonitoringandcontrolasonelevelofimportancewithrespecttotheoverallmodel.
Themodelisdesignedwiththreeprimarycomponents,eachlabelledaccordingtothelanguageofthemetacognitionliterature.Themodel’scorecomponentisregulation,whichatitscentreinvolvesamonitorand control loop. Because of the interrelated nature of monitoring and controlling, we tended toconceptualizetheseasoneloop,ratherthantwoindependentcomponents.Wenotethattheregulationcomponentalsocontains initiationandobjective sub-components, respectively labelledas triggerandgoal.Regulationiscontinuallyinteractingwiththeothertwocomponents:Knowledge,whichrepresentsmetacognitive knowledge (the storage of strategies anddecision information), and Experience,whichrepresentsmetacognitiveexperience(feelingsandaffectivecontribution).
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 29
Table1:Modelsub-componentsdescribedintermsofmetacognitionandreflection.
Wenotethatthemodelasvisualizedcouldinfercausalrelationshipsbetweenvariouscomponentsandthatthereisthereforeasenseinwhichatemporaldimensiontothemodelcanbeinferred.Thiswasnotourintentionasthecomponentsarecontinuallyinteractingwitheachotherthroughoutmetacognitiveandreflectiveactivity.Also,asweadoptedmetacognition inspiredlabelsforthecomponentview,therelationshipbetween themodeland reflectioncanbe lessobvious.Toassistwith these links,Table1drawsattentiontotheapplicabilityofthemodelforbothreflectionandmetacognition.
2.4 The Use of Reflective Writing
Anessential point is that the proposedmodel centres on the cognition of the learner, and thereforecannot be analyzed directly. An interfacemust be provided between this conceptualmodel and anylearninganalyticsthatwemighthopetoperform.Inthisstudy,wehaveusedreflectivewritingasthisinterface. Reflectivewriting is an existing, well-accepted learning activity, enabling this type of studywithout imposing a non-related task on the learning process.We acknowledge, of course, that othermeansofanalyzingmetacognitionexist(Koriat,2000).
Within the modern educational context introduced in Section 1, the requirement to move towardsteaching skills required for lifelong learning has resulted in increasing attention on reflectivewriting.Reflectivewritinghasthepotentialtodevelopmetacognitiveandreflectivecapabilitiesintheirownright(Reidsema,Goldsmith,&Mort,2010),andsohereweattemptedtoextractanalyticsspecificallydirected
Metacognition Reflection
Trigger(Regulation)
Aconsciousorunconsciouscognitiveevent,particularlyaproblemorincongruence.
Aconsciouscognitiveeventoranexternaleventthatneedsimprovementormodification.
Monitor(Regulation)
Monitoringofcognitiveprocesses,bothconsciouslyandunconsciously.
Consciousthinkingaboutmentalprocessesorexternalbehaviours.
Control(Regulation)
Utilizingpre-learnedstrategiestocontrolcognitiveprocesses.
Takingactiontomodifymentalprocesses,orexternalbehavioursbasedoninputinformation.
Goal(Regulation)
Resolutionofthetriggerproblemordissonance.
Successfulmodificationorimprovementoftriggerevent.
Knowledge Memorydedicatedtostoringmetacognitiveknowledgeinparticularstrategiesandtheirefficacy.Usedinthemonitor–controlloop.
Eitherinternalmemoryorexternalrecordingofthoughtsthatcanbeusedasnecessary.
Experience Affectiveimpactonmonitor–controlloop.Assistswithstrategyformationandevaluation.
Emotionalcontributiontomonitoringandcontrol.Assistswithestablishingpersonalvalueandsignificance.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 30
towardstheanalysisofmetacognitiveactivityasitarisesinreflectivewriting.Although,thismightappeara reasonable approach, wemust be careful when adopting it within a formal education setting. Therequirement to measure what has been learned in order to justify the importance of the learningexperiencecanhaveasignificantnegativeimpactonreflectivewriting.“Assessmentinvolvespresentingone’sbestwork,whereas reflection involvesuncertainquestioning, self-criticism,exploring, tryingoutideasandacknowledgingthemessynatureofreality”(Wharton,2012,p.490).Furthercomplicatingtheassessmentofreflectivewriting,Reidsemaetal.(2010)notethatitallowsstudentsto“engagewiththeirbeliefs,values,uncertainties,desiresandquestionsandtoclarifywhattheyknowandmoreimportantly,do not know about a situation” (p. 10). This can place reflective writing tasks in direct conflict withsummativeassessment,resultinginaconflictofparadigmsbetweenaninstitutionalneedforapositivist,product-driven perspective, as opposed to the learner’s needs for a constructivist process-drivenapproach(Ross,2011).Thisisoftenseenwhenthereisatendencytofocus“onwhathasbeenlearnedrather thanonhow ithasbeen learned,and theemphasis ison improving the reflectivewritingstyleratherthanonlearningaboutlearning(metacognition)”(Mair,2012,p.150).
Our approach avoids these issues by focusing on the discovery of metacognitive activity through alearner’swritingwithout requiring theuseof summative assessment. This approach couldpotentiallyprovideautomatedfeedbackderivedfromtheanalyticstothelearner.Usingthisapproach,metacognitiveactivityandreflectioncanbeencourageddirectlywithinthelearningprocess,ratherthanviaassessment.
3 REFLECTIVE WRITING ANALYTICS
WeconsiderReflectiveWritingAnalytics(RWA)tobetheanalysisofreflectivewritingandreportingofresultantinformationaboutthewritersandtheircontexts.ThecomputationalanalysisthatfollowsisRWAforthediscoveryofmetacognitiveactivityonthepartoflearners.
3.1 Data
The researchpresentedhere involved theanalysisof6,090student reflectionscapturedprogressivelyoverasemesterusingawebapplicationcalledGoingOK.1Sixhundredandfifty-sevenstudentsfromthreedifferentBachelor’sDegreeprogramsparticipatedinthestudy:athird-yearEnvironmentalEngineeringunit(DS-E);afirst-yearScienceunit(DS-S);andafirst-yearInformationTechnologyunit(DS-I).Foreachunit,theuseofthesoftwareandparticipationinthestudywasoptional,andthenumberofstudentswhowrotereflectionsusingthewebapplicationissummarizedinTable2.Ineachcase,reflectivewritingwasa requirement of the unit; however, the nature of this requirement and the extent to which it wasassessed, differed markedly. All data in this study is drawn from reflections written using the webapplication.
1GoingOKwaswrittenbyAndrewGibsonaspartofaQUTEducationresearchprojectthatcollectedthereflectionsofearlycareerteachers as they transitioned from student and developed their professional identities during their first year of teaching(http://GoingOK.org/)
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 31
Table2:Reflectivewritingdatasetsizes*
FullDataSet DS-E DS-S DS-I
Reflections N=6,090 188 740 5,162
Authors 657 35 145 477
Meanrefs/author 9.3 5.4 5.1 10.8
*“Reflection”isasingleentryinthewebapplication,“authors”arestudents,andthe“meanreflectionsperauthor(refs/author)”istheaveragenumberofreflectionswrittenbythestudentsoverthecourseofthesemester.3.2 Computational Analysis
Thecomputationalanalysiswasgovernedbytheconceptualmodel(Section2),withparticularemphasisonthekeyfeaturesidentifiedinthecomponentview.Ouraimwastoexploretheextenttowhichtheseconceptualfeaturesmaycorrespondtolexicalandstructuralfeaturesinthereflectivewriting.Ourchoiceoffeatureswasbasedonawiderangeofpreviousresearchidentifyingrelationshipsbetweentextfeaturesandhumanfactors. Inparticular,wedrewonPennebakerandChung’s (2011)workonpronouns,andRyan’s(2011)workonlinguisticfeaturesinacademicreflectivewriting,alongwitharangeofotherwork(Wharton,2012;Tang&John,1999;Reidsemaetal.,2010;Ullmann,Wild,&Scott,2012).
We linked the conceptual model to the reflective text indicators via an algorithm that progressivelymapped low-level grammatical features in the writing (posTags) through to higher level annotations(metaTags),whichwederivedfromthemodel.Thealgorithmthatwedevelopedcomprisedfourlevels:
• Partofspeech(POS)taggingofasentence(posTags)• MatchingofPOStagpatternstoidentifykeyphrases(phraseTags)• Matchingofphrasepatternstoidentifypotentialforannotation(subTags)• Filteringofmatchedphrasepatternstoselectfinalrelatedmodelcomponent(metaTags
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 20
ThealgorithmwasimplementedinScala.2POSTaggingwasimplementedusingtheFactorieframework(McCallum, Rohanimanesh, Wick, & Schultz, 2008) with a pre-trained parser model based on theOntoNotesEnglishcorpus.Whilethisisnotreflectivewritingspecific,itprovidesstateoftheartaccuracyinPOStaggingandhasprovedeffectiveforthepurposesofexploringthepotentialoftheapproach.
Informed by both the literaturementioned above and exploratory trials on the data,we selected 17phrasetagpatternsthatcapturedarangeofpotentiallymeaningfulphrases.Thesewerebasedonthe5generalPOSpatterns listed inTable3.ThefinalphraseTagswereobtainedbyfilteringthePOSpatternmatching results by a lexicon for each phraseTag.3 The filtering lexicons were derived by manuallyidentifying phrases from amongst the results from the POS pattern matching that showed potentialalignmentwith themodel.Thepronoun–verbandprepositionmatchingalsohavegeneral tags,whichincludedallphrasesnotspecifically filtered.Totunethealgorithm,the listsofgeneralphrasescanbesearchedforanyimportantphrasesmissedbythefilter.
AsecondmatchingpatternselectssubTagsbasedonthephraseTagpattern,andthenafinalmatchingpatterncombinesthesesubTagstoprovidethefinalmetaTagthatlinkswiththemodel.Thesepatternswereselectedusingtwocriteria:1)theindicativemeaningofthephrase,and2)theextenttowhichthe 2Examplecodecanbefoundathttp://nlytx.io/2016/metacognition3Thefullfilteringlistcanbefoundinthecodeonlineathttp://nlytx.io/2016/metacognition
Table3:POSpatternsfortheselectionofphraseTags
Pattern CorrespondingphraseTags
possessivepronounfollowedbyanyadjectivesandnouns
selfPossessive(e.g.,myteam)groupPossessive(e.g.,ourgroup)othersPossessive(e.g.,herproject)
pronounfollowedbyanyverbs,adverbs,conjunctions,andprepositions
consider(e.g.,wedecidedtogo)anticipate(e.g.,weneeded)emotive(e.g.,iamcomingalongreallywell)generalPronounVerb(e.g.,wehad)
prepositionfollowedbyrepetitionofanyPOS compare(e.g.,likeweare)temporal(e.g.,aftergetting)pertains(e.g.,withblooddonations)manner(e.g.,withoutdistractions)outcome(e.g.,fromourpracticepitchlastweek)generalPreposition(e.g.,athand)
modalfollowedbyanyverboradverb definite(e.g.,cantrust)possible(e.g.,wouldbefocusing)
personalpronounfollowedbyanyPOSandendingwithapersonalpronoun
selfReflexive(e.g.,ifeellikei)andgroupReflexive(e.g.,wetimedourselves)
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 21
phraseneededtobecombinedwithotherphrasesinordertosupportthecorrespondingmetaTag.ThefinalchoiceofpatternsisoutlinedinTable4.
Table4:RelationshipbetweenmetaTags,subTags,andphraseTags
metaTag subTag PhraseTagPattern
RegulationmonitorControlAND(triggerORgoal)
Trigger outcome
MonitorandControl temporalOR(pertainsANDconsider)
Goal anticipateORdefiniteORpossible
Knowledge selfPossessiveORcompareORmanner
Experience emotiveORselfReflexive
The progressive layering approach thatwe used in the algorithmwas beneficial in twoways. First, itenabledustomaintainconnectionsbetweenthephrasesinthetextandthecomponentsinthemodelbyprovidingclearlinksbetweeneachlayer.Second,thelinksenabledustotraceinaccuraciesintheanalyticsbylookingattheunderlyingpatterns.However,thesebenefitscomeatacost,andevaluatingeachlayercanbeverytimeconsuming,particularlywithlargeamountsofdata.Weseethisapproachasafirststep,allowingustogainanindicationofhowwellthemodelandthealgorithmworktowardsthediscoveryofmetacognitiveactivityfromthelearner’swriting.AnexampleoftheapplicationofourmappingalgorithmtoasentenceisillustratedinTable5,andfurtherexamplesareprovidedintheAppendix.
Table5:ExamplederivationofmetaTagtagsfromasentence.
Text Onseeing
thesheeramount
ofit , Ithink we lhave
tosplitupthework.
posTag INVBG DTJJNN INPRP , PRPVBP
PRP MDVB
TOVBRPDTNN.
match start(IN)&repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)
start(PRP)&repeat(RB,VB,CC,IN)
start(MD)&repeat(RB,VB)
filter pertains(startsWithAny(across,to,of,that,on,among,about,under,over,with,within,around,whether,for,in,e.g.,i.e.))
consider(containsAny(feel,felt,seem,think,realise,being,thought,decide,know,appeared,learn,experience,focus,found,guess,believe,wonder,find,personal,reflect,brain,understand,understood,notice,myself,recent,imply,record,probably,emotion,physical,trust,deal,barely,pretty,extremely,incredibly,miraculously,supposed))
definite(startsWithAny(will,’ll,ca))
phraseTag
pertains consider definite
match temporalOR(pertainsANDconsider) anticipateORdefiniteORpossible
subTag monitorControl goal
metaTag
regulation
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 22
3.3 Early Decisions
Anumberofkeydecisionsweremadeduringthecomputationalanalysis.Theseinvolvedquestionsonhowweapproachedtherawdata,andalsomoreimportantlyhowwewouldapproachthegranularityoftheresultsthatarosefromthealgorithm.
Thewebapplicationusedforcollectingthereflectivewriting(GoingOK)includestemporalinformationintheformoftimestamps.Anearlydecisionregardingthedatawastoignorethistemporal information.Althoughthetimestampdatacouldpotentiallybeusefulinshowingthedevelopmentofmetacognitiveactivityovertime,wefeltthatutilizingitforthisprojectwouldaddunnecessarycomplexitytotheanalysiswithoutprovidingsignificantvaluetoaddressingtheresearchquestions.Thus,ourattentionwasfocusedonevidenceofmetacognitiveactivityratherthanondevelopmentofthatactivity.
Initially,wehadalso consideredcombiningall reflections fora singleauthor intoonedocument.Thiswouldhavesignificantlyreducedthenumberofreflections(1perauthor)andforauthorswithverybriefreflections,itwouldhavepotentiallyprovidedahigherprobabilityforthedetectionoffeatures.Howeveritbecameapparentfromearlyobservationsofthedata,thatthemodelwewereseekingwasevidentinsingle individual reflections (even short reflections), and thatwe gainedmore information about theauthor’smetacognitiveactivitybykeepingthereflectionsseparateforanalysis,andthenconsideringtheresultant analysis as a whole with respect to each author, such as, for example, the frequency ofmetacognitiveactivity.
3.4 Categorization of Metacognitive Activity Asthisworkwasexploratoryinnature,wetookacautiousapproachwithrespecttothegranularityoftheanalysis. We settled on three categories of reflective writing: 1) reflections that showed significantevidenceofmetacognitiveactivity,termed“strong”reflections;2)reflectionsthatshowedverylittleornoevidenceofmetacognitiveactivity, termed“weak”reflections,and3) reflectionsthatwereneitherclearly strongnorweak, termed “undetermined.”Originally,weanticipated thatdistinctionsbetweenthesegroupswouldbereasonablyeasytomake,however,thevariabilityinthedatameantthatdecisionsregardingtheircategorizationweremorecomplexthanwefirstthought.
Weundertookaseriesoftrialstodetermineappropriaterulesforclassification.Toreducecomplexity,thesewereallundertakenonthesmallerDS-Edataset.Inwhatfollows,wewillexplainhowwearrivedattheheuristiccategorizationrules(listedinTable6)appliedtotheotherdatasets(seeTable7).Thiswillthenleadtofurtheranalysisandexplorationoftheresults.
Initialcategorizationwasbasedonthenumberofuniquemetacognitivetagsassignedtothereflection,assumingthatitcontainedaregulationtag.Thisrequirementwasappliedbecauseregulationisacorecomponentofthemodel,andsoalackofregulationwasexpectedtoidentifyaweakreflection.Ifthereflectionwastaggedwithallthreetags(regulation,knowledge,andexperience),thenitwasclassifiedasstrong.Weakreflectionswereidentifiedbyzerotagsorasingletagthatwasnotregulation.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 23
Weusedaprocesscalledanomalyrecontextualization(Gibson&Kitto,2015)torefinetheheuristicsofthecategorizationrules.Thisinvolvedrunningtheanalysiswithagivenrule,identifyinganomaliesintheresults,thenusingadditionalfeaturestoresolveorrecontextualizetheanomalies.Thefirstcategorizationresultedinanumberofanomaliesinboththestrongandweakcategories.Furtherexaminationofthesereflections revealed a correlation between the reflections’ sentence counts and the anomalies in theresults. Although the number of sentences did not directly correlate with metacognitive tagging, allanomalousreflectionsthatshouldhavebeenidentifiedasstronghadaminimumofthreesentences.Wewereabletousetheminimumthreesentencefeaturetoresolvetheanomalyanditwasaddedtotherules.
Itwas found inearlyexperimentation that thecomputationalanalysiswasbiased todeemreflectionscomprisingasmallnumberofsentencesas“strong.”ThiswasduetoametriccalledsubTagDensity,whichnormalizedtagcountstothenumberofsentencesbytakingthetotalnumberofsub-componenttagsanddividing themby thenumberof sentences.Weutilized thisbecauseof thesignificantvariation in thelengthofreflections.However,thishadtheadverseeffectofcausingsomereflectionswithaverysmallnumberofsentences(e.g.,2)andanormalnumberoftags(e.g.,4)toberatedhigherthanreflectionswithslightlyhighersentencecount,makingthemmorelikelytobecategorizedasstrong.Weaddressedthis issue bymodifying subTagDensity to divide by the log of the sentence count (instead of just thesentencecount),andbyascribingavalueof0 tosubTagDensity forany reflectionswith fewer than2sentences.
Aspartoftheinitialcategorizationtrialswehadalsoidentifiedreflectionsthatweredifficulttocategorizeandtaggedthemaccordingtotheirlikelihoodtobeascribedacategory.IfthereflectionswereveryclosetobeingcategorizedasstrongthealgorithmascribedatagofpossibleStrong,andiftheywereclosetobeing categorized as weak, then they were tagged as possibleWeak. We examined the resultantreflections and noted that the potential changes identified above would resolve these boundaryreflectionsinfavouroftheexpectedcategory.Consequently,wemadetheadjustmentsdescribedandthecombinedchangesresultedinafinalcategorizationalgorithmthatutilizedtheheuristicrulesshowninTable6.
Table6:Heuristicrules
Class Rules
Strong sentenceCount>=3ANDmetaTagCount=3 OR sentenceCount>=3ANDmetaTagCount=2ANDmetaTags(regulation)ANDsubTagDensity>=4
Weak NOTmetaTags(regulation) AND metaTagCount<=1ORsubTagDensity<=3
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 24
Table7showstheresultingreflectionquantitiesforeachdatasetafterclassification.ItispossiblethatthesmallnumberofstrongandweakcategorizationsfortheDS-Edatacomparedwiththeotherdatasetsmayproveproblematic,asthecategorizationalgorithmwasinitiallytunedontheDS-Edataset.Weexaminethispotentialproblemfurtherbelow.
Table7:Reflectionnumbersaftercategorization.
3.5 Sentence Level Check
Atthispoint,wewantedtocheckwhethertheheuristicruleswereworkingappropriatelywithdatamoregenerally.Wewereparticularlyinterestedinstrongclassificationsasthismostdirectlyaddressedtheaimoftheresearch.ThischeckwasperformedbycomparingthemetaTagsofindividualsentencesannotatedbythealgorithmagainstamanualcodingofthesentencesundertakenbyoneoftheauthorsofthispaper.AstherewereonlyfourreflectionsintheDS-Edataset,wedecidedtocode“strong”reflectionsfromtheDS-Sdataset.Howevertokeepthetaskmanageable,wecodedthesmallernumberof“weak”reflectionsfromtheDS-Edataset.Wealsolimitedthetotalnumberofreflectionsbychoosingreflectionsbyauthorwhereatleastthreeoftheirreflectionswerecategorizedinthecategorybeingchecked.Thatis,forthestrongclassification,wechoseanauthor’sreflectionswhereatleastthreewerecategorizedasstrong.
Theintuitionbehindthisapproachwasthatalthoughwewerecheckingatthesentencelevel,theaimwastodiscovermetacognitiononthepartoftheauthor,andthereforethesentencesshouldultimatelybeunderstoodinthiscontext.Byincludingallofaselectedauthor’sreflectionsweretainedagoodselectionofreflectionswithinanauthor-centredcontext.Wealsoexpectedthattakingthisapproachwouldassistwithmitigatingagainstwritingstylevariance.ExamplesentencescanbeseeninTable8.ThecheckisofthemetaTagascribedtothesentencethatoccurspriortothestrong/weakclassificationperformedatthereflectionlevel.
All % DS-E % DS-S % DS-I %
Strong 837 13.8% 4 2.1% 59 8.0% 774 15.0%
Weak 4,180 68.6% 160 85.1% 564 76.2% 3,456 67.0%
Undetermined
1,073 17.6% 24 12.8% 117 15.8% 932 18.0%
Total 6,090 100.0% 188 100.0% 740 100.0% 5,162 100.0%
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 25
Table8:Examplesentences.
Check Sentence MetaTag SubTag PhraseTag
DS-SStrongAuthors
truePos ComparedtoothergroupsIfeltasthoughweareunder-preparedandthisismostlikelyduetothefactthatwestartedwellaftermostothergroups.
regulationexperienceknowledge
monitorControlgoalexperienceknowledge
anticipateemotivegeneralPronounVerbcomparetemporalpertains
falsePos Sowestartedthisweek’smeetingfirstlybydelegatingjobs,Ihadprintedoffthetemplatesuppliedonlearningresourcesinwhichwecouldallocateajobtitletoeachmemberpermeeting,asIwasthescribelastweekwenominatedKeithtodoitthisweek.
knowledge triggergoalknowledge
anticipategeneralPronounVerbpertainsmanneroutcomepossible
trueNeg Duringthischatthewholegroupco-operatedandeachhadavaluedinputintothetask.
falseNeg NexttimeImightjustwaitforsomeoneelsetochangethediscussionbacktothetopicsoI’mnottheonlyone.
goal generalPronounVerbpertainspossible
DS-EWeakAuthors
truePos Alltheassignmentsaredrivingmecrazy,andIfeellost. experience experience
considergeneralPronounVerbselfReflexive
falsePos YesterdayIsubmittedmyGroundwaterreport! knowledge knowledge
selfPossessivegeneralPronounVerb
trueNeg Justreportedtheprojectprogresstothelecturer.
falseNeg Itisalwaysdifficulttoinitiateaproject,whereeverymoveseemsheavyanduncertain. generalPronounVerb
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 26
3.5.1CheckofmetaTagforStrongAuthors(DS-SDataset)SelectingfromtheDS-Sdataset,basedonauthorswiththreeormorestrongreflections,yieldedatotalof272sentencesforfourauthors.Wemanuallycodedthedataforfalsepositives,falsenegatives,truepositives,andtruenegatives.Table9showsthat,overall,78%oftheclassificationswerefoundtomatch,with22%notmatching,themajorityofthesebeingfalsenegatives.
Table9:DS-SstrongauthormetaTagevaluation
Positive Negative Total %
true 55 156 211 78%
false 15 46 61 22%
Further analysis of thenon-matching classificationswas conducted todeterminehow thesemightbeavoided.Forthefalsepositives,wefoundthatnineofthe15relatedtoametaTagofknowledge,andcould have been classified as true negatives if the trigger subTag was used to prevent tagging asknowledge.Wealsofoundthatthiswouldresultinnoadditionaltruepositives.Inaddition,fiveofthefalsepositiveswerebasedontheregulationmetaTagandalloftheseinstanceswerebracketedbytruenegative sentences. This suggests these falsepositives couldbeaddressedby taking intoaccount theclassificationofthesentencesbeforeandafter,whichalignswithhowpeopletendtoreflect,becauseanideaismorelikelytobeencapsulatedinanumberofsentencesratherthanjustone.
Forthefalsenegatives, thereappearedtobestrongcorrelationswiththesubTags,providingaway inwhichmanyofthefalsenegativescouldbeclassifiedastruepositives.Inviewofthis,weconsideredthatwemayneedtoaddressthewayregulationisarrivedatfromtheunderlyingsubTags.Howeverwealsonotedthatthiscouldhaveresultedinalargeincreaseinfalsepositives.Wefoundthat17ofthe46wouldhavebeentruepositivesifthegoalsubTagwasused,fourifthetriggersubTagwasused,and14ifthepertains phraseTag was used. The numbers provide an indication that this area of the classificationprocessrequiresfurtherrefinement,andthattheapproachwillneedtobemorecomplexthanpurelyutilizingthesubTagsandphraseTags.
Whencheckingthedata,wewerenotsurprisedtofindthatauthorstendedtowardaparticularstyleofreflection. We thought that these stylistic differences may have had a material impact on theclassification,soweseparatedtheresultsforeachauthor(seeTable10).Thiscouldbeanareaforfurtherinvestigationaspartofamoreauthor-focusedanalysis;however,forthisstudywefoundtheresultstobelessvariablethanexpected,anddidnotpursuethislineofenquiryfurther.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 27
Table10:Resultsbyauthor.
Author1 % Author2 % Author3 % Author4 %
falsepositives 11 0.1% 2 0.5% 0 2.0% 2 1.2%
falsenegatives 23 0.9% 11 0.5% 3 0.0% 9 0.6%
truepos&neg 121 1.9% 41 2.7% 15 2.0% 34 2.6%
35 1.0% 15 1.0% 6 20.0% 15 10.8%
3.5.2CheckofmetaTagforWeakAuthors(DS-EDataset)Initiallyweselectedreflectionsfromauthorswiththreeormorereflectionscategorizedasstrong.Werecognizedthepossibilitythatthealgorithmmayshowsignsofsuccesswiththisselectionifover-fittingwasoccurringwithinthehigherqualitydata.Therefore,wefeltthatweneededtoundertakeasimilarevaluationwith sentences fromauthorswithmostlyweak category reflections. That is, a selectionofreflections based on authors with three ormoreweak reflections.We used the DS-E dataset, whichyieldedatotalof437sentencesfor22authors.Likethepreviouscheck,wemanuallycodedthedataforfalse positives, false negatives, true positives, and true negatives (see Table 11). Overall, we wereencouraged by finding very similar results to the previous study, with 80% matching, and 20% notmatching (comparedwith78%and22%respectively fromthestrongauthors).Adifferentiating factorwith the weak author reflections was that most mismatches were false positives (rather than falsenegatives).Thismakessense,asitsuggeststhatouralgorithmispossiblynotasaggressiveasitcouldbe.Amoreaggressivealgorithmwouldlikelyresultinareductionoffalsenegativesinthestrongreflectionsaswellasreducethenumberoffalsepositivesintheweakdata.
Table11:DS-EweakauthormetaTagevaluation.
Positive Negative Total %
true 53 296 0 80%
false 61 27 349 20%
Ofthe61falsepositives,42belongedtoreflectionsclassifiedasweak,meaningthatthefalsetaggingofthesesentencesdidnotmakeamaterialdifferencetotheoutcome.However,40ofthesesentencesweretaggedwithknowledge,indicatingthatifthiseffectappliedtoallreflections,thenfewerreflectionsmaybetaggedasweak.Amongtheother19falsepositives,12werepartofundeterminedreflections,whichcouldpossiblybecomeweak iftheclassificationwas improved.Moreofan issuearosewiththesevenfalsepositivesamongreflectionsclassifiedasstrong.ThesewereallbasedonaregulationmetaTag,andbecausethisisacriticalcomponentofastrongclassification,allsevenofthesereflectionswouldhavebeenclassifiedasundeterminedifthesefalsepositiveswerefixed.Theseissuessuggestedtousthatthesideeffectsofthefalsepositivesonstrongclassificationissignificantandworthyoffurtherattention.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 28
Withtheexceptionoftwosentences,thefalsenegativeswerepartofreflectionsclassifiedasweak,soweneededtoassesstheextentoftheircontributiontothefinalreflectioncategorization.Inourassessment,allbutoneofthesesentencesshouldhavebeenclassifiedasexperience,andnoneoftheothersentencesinthereflectionsincludedaregulationtag,sotherewasnomaterialchangetothereflectionclassification.Similarlywithonethatcouldhavebeentaggedasregulation,therewerenoothersentencesinthesamereflectiontaggedasknowledgeorexperiencesoastoinappropriatelypromotethereflectiontothestrongcategory.Thissuggeststousthatdespitethefalsenegatives,thereisamoderatingeffectinthealgorithmthatarisesbytakingtagsfromallsentencesindeterminingthereflectionclassification,whichappearstobeagoodthing.
3.5.3AlgorithmChangesFrom the checks, we determined that only two changes to the algorithm would not have adverselyaffectedtheevaluationdata.Bothofthesechangesrelatedtothestrongauthorsentences.
The first was to consider the relationship between the trigger subTag and the knowledge metaTag.Lookingattheoriginalphrasepatternmatching,wededucedthattheissuemighthavepresentedduetothesimilaritybetweentheunderlyingphraseTagsofoutcome(relatedtotrigger)andcompare(relatedtoknowledge).ItappearedthatitwouldbebettertohaveincludedcomparetogetherwithoutcomeforthetriggersubTag;however,wedeterminedthatfurtheranalysisisrequiredtoascertainthatthisdoesnotadverselyaffectotherclassifications.WealsonotedthatissueswiththeknowledgemetaTagwerealsopresent in theweakauthordata.This reinforces the fact that thecompositionof this tagneedsmoredetailedexamination.
Thesecondchangeinvolvedmoderatingregulationtaggedsentencesbasedonsentencesbeforeorafter.That is, ifthesentencebeforeorafterhasnometaTag,thenremovetheregulationmetaTag.Wealsodeterminedthatitwouldbebesttonotimplementthischangeuntilitseffectshadbeenexaminedwithinalargerdatasettodeterminewhethertherewouldbeanyadversesideeffects.
3.6 Metacognitive Analysis by Author
OurfinalanalysisforthisstudywasonDS-I,thelargestofthethreedatasets.Whilethesizeofthedatasetdidnotpresentanycomputationalproblems,itdidmakeourassessmentoftheresultsproblematic.Thelackofpre-annotatedreflectivetextcorporamakesitdifficulttoundertakehigh-qualityevaluations,andalthoughwecouldchoosesmallersubsetsofDS-EandDS-Stomanuallyannotate,theDS-Idatasetwasnotquiteasstraightforward,asitdidnothaveaclearsubsettoworkwith.However,thelargersizeofDS-Idoeshavesomebenefits;repeatedpatternscanbecomemoreobvious,ascananomaliesinthedata.Ourapproachwastoexaminethedataasawholeforobviouspatternsandanomalies.
Inlinewiththeoverallresearchobjectiveofidentifyingmetacognitiveactivityinthelearner,wedecidedtotakeanauthor-centricviewofthedata.Thefirsttaskweaddressedwasawaytoranktheauthors.Wewantedourrankingalgorithmtoranktheauthorsexhibitingthemostmetacognitiveactivityhighest,andauthorsexhibitingtheleastactivitylowest.However,weknewthatauthorsusuallyhadamixofstrong,
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 29
weak,andundeterminedreflections,andwealsoknewthattherewasawidevarianceinthelengthandnumberofreflections.Wealsoknewthat,onaverage,thereweremorethanfourweakreflectionstoeverystrongreflection(seeTable7).Inordertoaddressthesefactors,wesettledonanalgorithmthatweightedstrongreflectionsas+2andweakreflectionsas-1.Undeterminedreflectionswereweightedwith0.Wesummedtheweightingsofallreflectionsofanauthoranddividedtheresultbythelogofthetotalnumberofreflections.
This algorithm proved useful in ranking the authors, and on running itwewere immediately able toidentifytwokeyanomalies:thehighestrankedauthorandthelowestrankedauthor.Bothrankingsoftheseauthorswereoutsidethealmostsymmetricrangeoftheother475authors(-7.44to+7.82)withthehighestrankedauthorat+9.232andthelowestrankedauthorat-27.126.Whatwasinterestingaboutthese two authors, however, was the significant difference in their reflections to the closest rankedauthors.
Thehighestrankedauthorwroteverylongreflections,thelongestbeing53sentencesandtheaveragebeing25.9sentences.Whilesomeauthorswrotelongreflections,theytendedtowriteveryfew,whereasthisauthorwrote15reflectionswithonly fiveof themhavingfewerthan20sentences.However, thelargeamountoftextforthisauthoralsohighlightedpatternsinthephraseTagsthatwerenotimmediatelyobvious with smaller reflections. For example, there appeared to be a very large number ofgeneralPronounVerbtags, indicatingthat related filteringmaybetooaggressiveandthatmorewordswereneededintheassociatedlexiconstominimizephrasesfallingthroughtothegeneraltagratherthanbeingcaughtbymorespecifictags.ThelongertextsalsorevealedanunusuallyhighnumberofpertainphraseTags,oftenfourtimesgreaterthanthenexthighesttag.Thiscouldmeanthatthefilterforpertainsisnotaggressiveenoughandneedstobetightenedup.Finally,weobservedthatgoalisoftenthemostfrequentsubTagandthatitoftenco-occurswithtrigger,butregulationislessfrequent.ThisindicatesthateitherweeitherneedtoimprovemonitorControldetection,orweneedtorelaxtherulefortheregulationmetaTag.
Attheotherendoftherankings,thelowestrankedauthorwrotealotofveryshortreflections,145intotal,ofwhich113wereonlyonesentencelong.Unlikethehighestrankedauthor,wedidnotfindanyobviousimplicationsforchangingthealgorithmusingthisauthor’sreflections.
Basedonthesefindings,wesettherangeofrankingsfrom-8to+8,whicheffectivelyexcludedthesetwoanomalousauthorsfromthedataset.TheresultsforauthorsinthisrangeareprovidedinTable12.
Aninterestingfindingfromtheseresultsisthecomparisonofthethreeclassificationsovertherankingrange.Figure3showsthattheundeterminedclassificationappearstohaveagreatereffectontheweakclassificationthanonthestrong,andthatthiseffectstrengthensinthetophalfoftherange.Thissuggeststhatwemightneed to fine-tune theclassificationalgorithmaround theboundarybetweenweakandundetermined. This is not surprising, as the algorithm is looking for the presence of patterns whenclassifyingasstrong,resultinginarelativelyclearboundarycondition,butitistheextentoftheabsence
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 30
ofthosepatternsthatresultsinthedivisionbetweenweakandundetermined.Wemayneedsomemorepositiveindicatorsoflackofmetacognitiveactivityinordertostrengthenthealgorithminthisarea.
Table12:DS-Sanalysisbyranking.
Figure3:PercentageofmetaTagsforeachrankingrange.
Bottom Lower Middle Upper Top All
Rank -8.0to-4.7 -4.69to-3.1 -3.09to-0.01 0to2.22 2.23to8.0 -8.0to8.0
Authors 47 100 173 106 49 475
Reflections 876 1,161 1,621 785 559 5,002
Meanrefs 20.33 12.99 11.6 12.7 13.59 13.85
Minrefs 12 5 3 1 3 1
Maxrefs 33 31 26 27 25 33
StdDev 6.06 4.62 5.11 6.12 4.6 6.12
Strong 16 24 179 238 304 761
Weak 790 991 1,043 359 137 3,320
Undeterm. 70 146 399 188 118 921
Strong% 1.8% 2.0% 11.0% 30.3% 54.4% 15.2%
Weak% 90.2% 85.4% 64.3% 45.7% 24.5% 66.4%
Undeterm.% 8.0% 12.6% 24.7% 24.0% 21.1% 18.4%
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 31
Alsoofinterestintheseresultsweretheminimumandmaximumnumberofreflectionsforeachrankingband.Overall,lowerrankedauthorswrotemorereflectionsthanthoseinthehigherranking.Thissuggeststhatmorewritingmaynotnecessarilyprovidebetterevidenceofmetacognitiveactivity.Therelationshipbetween the number of reflections and overall metacognitive activity could benefit from furtherinvestigation,asthismayhaveimportantimplicationsfortheteachingofreflectivewriting.
4 IMPLICATIONS FOR LEARNING ANALYTICS
WecontendthatourworkholdsanumberofimplicationsforthefieldofLearningAnalyticsthatwouldbeworthyoffuturepursuit.
First,anumberoffindingsarisingfromthedevelopmentoftheconceptualmodelcouldimpactthefield.Wefounditessentialtoclarifytherelationshipbetweenmetacognitionandreflectionpriortotheworkonthecomputationalalgorithm.Weseethisclarificationasafirststep,withalotofpotentialforfurtherdevelopmentofthemodel,particularlyaroundtheexpansionofthemodelintotheexternal/socialarea.
Asourinterestwasfundamentallyonmetacognitiveactivity,wedidnotexaminethevariousmodelsofreflectivewriting,northerelationshipsbetweenthemandthecognitiveactofreflection.Webelievethatthe conceptual model would benefit from further work in this area. A greater understanding of therelationships between cognition and writing may also inspire new approaches to reflective writinganalytics.
Thelackofanannotatedcorpusofreflectivewritingturnedouttobeasignificanthandicapinundertakingthisstudy.Weareconsciousthatafullandproperevaluationofthevariousheuristicswasnotundertaken,andthatbeingabletotestthealgorithmsagainstapre-existingannotateddatasetwouldhavebeenasignificantbenefit.However,wealsonotethatthelackofthistypeofresource,andtheethicalissuesincreatingone,offersanopportunityforthedevelopmentofnewapproachestoevaluatinganalytics.Onesuchmethodcoulduseautomatedfeedbackasamechanismtocollectvalidationfromtheauthor;thatis,providinginformationbacktoauthorsabouttheirreflectionsandrequestingaresponseastotheextenttowhichitisaccurate,reasonable,fair,useful,etc.Thiswouldenableanautomatedannotationofthecorpus,whichcouldbeusedtotrainmachinelearningmodelsforcomputationalanalysis.
Relatedtothisidea,istheautomatedgenerationofformativefeedback.Fromthereflectiondata,wecaneasilyidentifykeydomainphrasessuchas“mygroup,”“ourproject.”Wecanalsoidentifydeficienciesinthetypeofreflection,suchasalackofexpressionofpersonalfeelings.Puttingthesetwoaspectstogetherwouldallowustogeneratefeedbackalongthelinesof“Perhapsyoucouldwriteabouthowyoufeelaboutyourgroup.”
Finally,asidentifiedabovewithrespecttotheconceptualmodel,reflectionisneitherrestrictedtowritingnortobeingexpressedlinguistically.Weconsiderthecomputationalanalysisofnon-linguisticreflectionto be a particularly interesting area for future research, with potential benefits in applying learninganalyticstosportsperformance,musicalimprovisation,andothernon-linguisticlearningactivities.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 32
5 CONCLUSION
Inthisstudywehaveexaminedarangeofconceptualizationsofmetacognitionandreflectionandbroughtthemtogetherintoaunifiedmodelforthepurposeofprovidingclarityastohowthefeaturesmaybeanalyzed computationally from reflective writing. Based on this conceptual model, an algorithmwasdevelopedthatcategorizesreflectionsbasedonmetacognitiveactivity.5.1 Limitations
Becauseoftheexploratorynatureofthisstudy,threesignificantlimitationsshouldbeaddressedinfuturework.First,wedidnotconductafullempiricalstudyofthealgorithmssotheresultsindicatedinthispaperareindicativeofthepotentialoftheapproachratherthanaconfirmationofitsvalidity.Weacknowledgethatinorder toestablish thealgorithms’ truepotential to the field,anumberofadditional considerationswouldneedtobetakenintoaccount.Inparticular,wewouldconsiderthefollowingnecessary:1)theuseofdifferentdatafortheevaluationthanthatusedtodevelopthealgorithm;2)manualannotationoftheevaluationdatawithmultiplehumanannotatorstoestablishinter-raterreliability;3)makingexplicitanydifferencesbetweenannotationcriteriaandalgorithmcriteria;and4)wherejustified,theapplicationofconventionalstatisticalanalysistotheresults.
Second,thedatasetusedforthisstudywasnotnecessarilyrepresentativeofreflectivewritingingeneral.Itinvolvedthecollectionofmostlybrief,unstructuredreflectionsonaregularbasisoveraperiodoftime.Thistypeofreflectionmaybesignificantlydifferentthanthelonger,morestructuredreflectivewritingtypicalofstudentassignments.Theeffectofthesedifferencesontheperformanceofthealgorithmisunknown.
Third,thecombinationofstructuralpatternsandlexicalfilteringnecessarilymeansthatsometextwillbeomitted or inadvertently included, depending on the parameters and underlying lexicons. Weacknowledge much room for improvement in disambiguation, pattern matching, and morecomprehensivelexiconsandfilteringprocesses.
5.2 Future Work
Theselimitationsprovidesomekeyareasforfuturework.Weseevalueinafullempiricalstudybasedontheapproachoutlined inthispaper.Weseepromise inboththeconceptualmodelandthealgorithmdesign,andbelievethatamorecompletestudywithrigorousevaluationwilladvanceboththeconceptualandappliedaspectsofthiswork.Wealsoseepotentialinevaluatingdifferentstylesofreflectivewriting;inparticular, investigatingthedifferencesbetweenshort, recurrentreflectionand longer form,single-instancereflection.Thealgorithmitselfpresentsmanyopportunitiesforfurtherrefinement,andwouldbenefit fromthe incorporationofnovel techniques for lexiconexpansionandpatterndisambiguation.
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 33
Anotherpotentialavenueofresearchisintheareaofgeneratingformativefeedbackfortheauthor.Thisisalargeandcomplexspacethatthisstudyhasonlyhintedat.
5.3 Final Remarks
Overall, the approach presented here to discovering learner metacognition through the analysis ofreflectivewritingshowsdemonstratedpotential,andweconsideritworthpursuinginfurtherresearch.Whilethiscouldbeuseddirectlyforthepurposeofgeneratingreflectivewritinganalytics,weidentifiedpotential for significant impact in the fieldof learninganalyticsbydeveloping the software further toincludefeedbacktothelearner.Morebroadly,thispaperisaninitial,yetsignificant,steptowardsaformofanalyticscentreduponmetacognitionandreflection.Wehavepresentedawayinwhichthedetectionof metacognitive activity in an author of reflective texts might be automated, and shown how thisapproachmightbeusedasaformativefeedbacksteptoencouragemetacognitivethinkinginstudents.Weconsiderthistobeakey21stcenturycompetency,andholdthatitisessentialthatwefindwaystonurturethisskillinourlearners.Itisourambitionforthispapertoopenupafruitfulavenueforresearchinthisdirectionoflearner-centredlearninganalytics.
REFERENCES
Amsel,E.,Klaczynski,P.A.,Johnston,A.,Bench,S.,Close,J.,Sadler,E.,&Walker,R.(2008).Adual-processaccount of the development of scientific reasoning: The nature and development ofmetacognitive intercession skills. Cognitive Development, 23(4), 452–471.http://dx.doi.org/10.1016/j.cogdev.2008.09.002
Boud,D.,Keogh,R.,&Walker,D. (Eds.), (1985).Reflection:Turningexperience into learning. London:KoganPage.
Committee for Economic Development of Australia (CEDA). (2015). Australia’s future workforce?Retrievedfromhttp://www.ceda.com.au/research-and-policy/policy-priorities/workforce
Cornford,I.R.(2002).Learning-to-learnstrategiesasabasisforeffectivelifelonglearning.InternationalJournalofLifelongEducation,21(4),357–368.http://dx.doi.org/10.1080/02601370210141020
Dewey,J.(1916).Democracyandeducation.NewYork:TheMacmillanCompany.Efklides,A.(2008).Metacognition:Definingitsfacetsandlevelsoffunctioninginrelationtoself-regulation
and co-regulation. European Psychologist, 13(4), 277–287. http://dx.doi.org/10.1027/1016-9040.13.4.277
Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature ofintelligence(pp.231–236).Hillsdale,NJ:Erlbaum.
Flavell, J. H. (1978). Metacognitive development. In J. M. Scandura & C. J. Brainerd (Eds.),Structural/process theories of complex human behavior. Alphen a. d. Rijn, The Netherlands:Sijthoff&Noordhoff.
Flavell, J. H. (1979).Metacognition and cognitivemonitoring: A new area of cognitive-developmentalinquiry. American Psychologist, 34(10), 906–911. http://dx.doi.org/10.1037/0003-066X.34.10.906
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 34
Gibson,A.,&Kitto,K.(2015).Analysingreflectivetextforlearninganalytics:Anapproachusinganomalyrecontextualisation.Proceedingsofthe5thInternationalConferenceonLearningAnalyticsandKnowledge(LAKʼ15),275–279.http://dx.doi.org/10.1145/2723576.2723635
Herber,H. L.,&Nelson-Herber, J. (1987).Developing independent learners. JournalofReading,30(7),584–588.
Koriat, A. (2000). The feeling of knowing: Some metatheoretical implications for consciousness andcontrol.ConsciousnessandCognition,9,147–171.http://dx.doi.org/10.1006/ccog.2000.0433
Lehmann,T.,Hähnlein,I.,&Ifenthaler,D.(2014).Cognitive,metacognitiveandmotivationalperspectivesonpreflectioninself-regulatedonlinelearning.ComputersinHumanBehavior,32(C),313–323.http://dx.doi.org/10.1016/j.chb.2013.07.051
Mair,C.(2012).Usingtechnologyforenhancingreflectivewriting,metacognitionandlearning.JournalofFurther and Higher Education, 36(2), 147–167.http://dx.doi.org/10.1080/0309877X.2011.590583
McCallum, A., Rohanimanesh, K., Wick, M., & Schultz, K. (2008). FACTORIE: Efficient probabilisticprogrammingforrelationalfactorgraphsviaimperativedeclarationsofstructure,inferenceandlearning.PaperpresentedattheNIPS*2008WorskshoponProbabilisticProgramming,Whistler,Canada.Retrievedfromhttps://people.cs.umass.edu/~sameer/files/factorie-nipsws08.pdf
Mezirow,J.(1990).Fosteringcriticalreflectioninadulthood:Aguidetotransformativeandemancipatorylearning.SanFrancisco,CA:Jossey-Bass.
Moon,J.A.(1999).ReflectioninLearningandProfessionalDevelopment:Theoryandpractice.NewYork:PsychologyPress.
Paul,R.(1990).Towardsanewmeasureofsuccess:Developingindependentlearners.OpenLearning:TheJournal of Open, Distance and e-Learning, 5(1), 31–38.http://dx.doi.org/10.1080/0268051900050106
Pennebaker,J.W.,&Chung,C.K.(2011).Expressivewriting:Connectionstophysicalandmentalhealth.InH.S.Friedman(Ed.),Oxfordhandbookofhealthpsychology.NewYork:OxfordUniversityPress.
Quirk,M.E.(2006).Intuitionandmetacognitioninmedicaleducation.NewYork:Springer.Reidsema,C.,Goldsmith,R.,&Mort,P.(2010).Writingtolearn:Reflectivepracticeinengineeringdesign.
Paper presented at the 9thAnnual American Society for Engineering Education (ASEE)GlobalColloquium on Engineering Education, 18–21 October 2010, Singapore. Retrieved fromhttps://www.academia.edu/501660/Writing_to_Learn_Reflective_Practice_in_Engineering_Design
Ross,J.(2011).Tracesofself:Onlinereflectivepracticesandperformancesinhighereducation.TeachinginHigherEducation,16(1),113–126,http://dx.doi.org/10.1080/13562517.2011.530753
Ryan,M.(2011).Improvingreflectivewritinginhighereducation:Asocialsemioticperspective.TeachinginHigherEducation,16(1),99–111.http://dx.doi.org/10.1080/13562517.2010.507311
Ryan,M. (2013). The pedagogical balancing act: Teaching reflection in higher education. Teaching inHigherEducation,18(2),144–155.http://dx.doi.org/10.1080/13562517.2012.694104
Sanders,J.(2009).Theuseofreflectioninmedicaleducation:AMEEGuideNo.44.MedicalTeacher,31,685–695.http://dx.doi.org/10.1080/01421590903050374
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 35
Schön,D.A.(1983).Thereflectivepractitioner:Howprofessionalsthinkinaction.NewYork:BasicBooks.Takanishi,R.(2015).Dooureducationsystemsdoenoughtoenablelearnerstoflourishasindependent,
autonomous and well-balanced individuals? European Journal of Education, 50(2), 123–127.http://dx.doi.org/10.1111/ejed.12113
Tang,R.,&John,S.(1999,December).The‘I’ in identity:Exploringwriter identity instudentacademicwritingthroughthefirstpersonpronoun.EnglishforSpecificPurposes,18,Supplement1,S23–S39.http://dx.doi.org/10.1016/S0889-4906(99)00009-5
Ullmann,T.D.,Wild,F.,&Scott,P.(2012).Comparingautomaticallydetectedreflectivetextswithhumanjudgements. In A.Moore, V. Pammer, L. Pannese,M. Prilla, K. Rajagopal,W. Reinhardt, T.D.Ullmann, C. Voigt (Eds.), Proceedings of the 2nd Workshop on Awareness and Reflection inTechnology-Enhanced Learning (AR-TEL ʼ12), Saarbrucken, Germany (pp. 101–116). Retrievedfromhttp://ceur-ws.org/Vol-931/paper8.pdf
Wharton, S. (2012). Presenting a united front: Assessed reflective writing on a group experience,Reflective Practice: International and Multidisciplinary Perspectives, 13(4), 489–501.http://dx.doi.org/10.1080/14623943.2012.670622
Wilson, M. D. (2015). Pracademia: The future of the lifelong learner. About Campus, 20(2), 28–31.http://dx.doi.org/10.1002/abc.21189
Zimmerman,B.J.(2002).Becomingaself-regulatedlearner:Anoverview.TheoryintoPractice,41(2),64–70.http://dx.doi.org/10.1207/s15430421tip4102_2
(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3
ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 36
APPENDIX
Alivedemonstrationversionofthesoftwarecreatedaspartofthisstudy,aswellaslinkstothesourcecodeandvarioustechnicaldocuments,canbefoundat:http://nlytx.io/2016/metacognitionAdditionalexamplesfortheprocessinvolvedinderivingthemetaTagsfollow:Example1Text IknowI cancount on[name]todo
herbitandIknowshe willdo itwellasthe
recoverypartisnotnews
tous,EnviroSciencechicks.
posTag PRPVBPRP MDVB INNNTOVBPRPNNCC
PRPVBPRP MDVB PRPRBINDTNNNNVBZRBNN
TOPRPNNPNNPNNS
match start(PRP)&repeat(RB,VB,CC,IN)start(PRP)&any&end(PRP)
start(MD)&repeat(RB,VB)
start(IN,TO)&repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)
start(PRP)&repeat(RB,VB,CC,IN)
start(MD)&repeat(RB,VB)
start(PRP)&repeat(RB,VB,CC,IN)
start(IN,TO)repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)
filter containsAny(know)
startsWithAny(can)
startsWithAny(on)
containsAny(know)
startsWithAny(will)
containsAny(well)
startsWithAny(to)
phraseTag ConsiderselfReflexive
definite pertainsothers-Possessive
consider definite emotive pertains
match temporalOR(pertainsANDconsider)anticipateORdefiniteORpossibleemotiveORselfReflexive
temporalOR(pertainsANDconsider)anticipateORdefiniteORpossibleemotiveORselfReflexive
subTag monitorControl,goal,experience monitorControl,goal,experience
metaTag regulation,experience
Example2 Text Theyallseemed likelovelypeople andareinterested inparticipatingto
achievethebestpossiblemark.
posTag PRPDTVBD INJJNNS CCVBPJJ INVBGTOVB DTJJSJJNN
match start(IN,TO)repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)
start(IN,TO)repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)
filter startsWithAny(like) startsWithAny(in)
phraseTag
compare pertains
match selfPossessiveORcompareORmanner
subTag knowledge
metaTag knowledge