an initial evaluation of metacognitive scaffolding for experiential training simulators
DESCRIPTION
Slides from my presentation at EC-TEL 2012 in Saarbrucken, Germany, 18th September Abstract: This paper elaborates on the evaluation of a Metacognitive Scaffolding Service (MSS), which has been integrated into an already existing and mature medical training simulator. The MSS is envisioned to facilitate self- regulated learning (SRL) through thinking prompts and appropriate learning hints enhancing the use of metacognitive strategies. The MSS is developed in the European ImREAL (Immersive Reflective Experience-based Adaptive Learning) project that aims to augment simulated learning environments throughout services that are decoupled from the simulation itself. Results comparing a baseline evaluation of the ‘pure’ simulator (N=131) and a first user trial including the MSS (N=143) are presented. The findings indicate a positive effect on learning motivation and perceived performance with consistently good usability. The MSS and simulator are perceived as an entity by medical students involved in the study. Further steps of development are discussed and outlined.TRANSCRIPT
![Page 1: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/1.jpg)
An Initial Evaluation ofMetacognitive Scaffolding for
Experiential Training Simulators
Marcel Berthold, Adam Moore, Christina Steiner, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly,
Gary Donohoe, Gordon Power, Owen Conlan
![Page 2: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/2.jpg)
• Who?• ImREAL Project• Evaluating Self-Regulated Learning and Metacognitive Scaffolding in Experiential
Simulator
• How?• Cross cohort comparison
• Baseline usage of sim vs scaffolded usage• Initial – no cross correlation / matching• Third year medical students
• What (did we find out)?• Performance• SRL behaviour• Scaffolding encounters & effects• Analysis of reflection text New! (Not in paper)
Overview
![Page 3: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/3.jpg)
• create effective virtual reality training simulation tools that adapt to trainees’ past experiences or preconceptions
• closing the gap between the ‘real-world’ and the ‘virtual-world’
• Services should respond to users’ behaviour and adapt accordingly to the user model based on a pedagogical model
• Create services that can be connected to different simulators
Rationale of ImREAL
![Page 4: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/4.jpg)
What do we want to know?
1. Is self-regulated learning supported?
2. Does the simulator augmentation through the service lead to better learning performance?
3. Does the simulator augmentation through the service increase motivation?
4. Is the service well integrated in the simulation and learning experience?
![Page 5: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/5.jpg)
• SRL is composed of three cyclic learning phases: Forethought, Learning, Reflection (Zimmermann, 2002)
• Good SR learners use appropriate learning strategies and techniques
• Good SR learners achieve
better learning results and
are more motivated to learn
(Zimmerman, 2002; Veenmann, 2011)
SRL@ET workshop Kato Galatas, 15-06-2012
Self-Regulated Learning (SRL)
![Page 6: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/6.jpg)
• Integrated SRL cycles + Peer experience
(Hetzner et al., 2011)
Pedagogical framework in ImREAL
![Page 7: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/7.jpg)
Simulator
• Dialogic• Training for diagnostic interviews for psychiatric patients
![Page 8: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/8.jpg)
Reflection Elicitation / Scaffolding
• Simulator has pre-existing reflection tool
• Can be triggered by learner• Prompts to reflect also added by
instructional designer
• In practice mode, cohort 2 students had ImREAL Metacognitive Scaffolding Service (MSS)
![Page 9: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/9.jpg)
Phase 1 Trial
Scaffolding – Thinking Prompts
![Page 10: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/10.jpg)
• Approximately 140 (cohort 1 = 131 / cohort 2 = 143) medical students, on average 21 years old (40% male vs. 60% female, 80% Irish). They performed the simulation as part of their third year medical training at Trinity College, Dublin (cohort 1 in 2011, cohort 2 in 2012).
• Cohort 2 additionally has scaffolding prompts in practice mode only
• Experience• No experience with ETU simulator• Experienced with interviews (97 %)• limited experience with interviewing psychiatric patients (15 %)
Cohorts
![Page 11: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/11.jpg)
User
Reflection postings
Questionnaire Data
Log-Data
Assessment
Have you set a learning goal?
Methods
![Page 12: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/12.jpg)
• Project evaluation question
o Can SRL be enhanced through
Metacognitive Scaffolding Services (MSS)?
• Formative evaluation approach
• Research foci of ImREALMSS
Integrated ImREAL services
• Investigate: Impact on SRL
reports, behaviour, qualitative
feedback
http://www.empowertheuser.ie
Methods: Evaluation approach in ImREAL (First User trail)
![Page 13: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/13.jpg)
• No change in SRL compared to baseline• Highest usage of elaboration strategies
Memorizing strategies
Elaboration strategies
Organisation strategies
Planning strategies
Self-monitor-ing strategies
Time man-agement strategies
Achievement motivation strategies
Internal attri-bution
Effort strate-gies
First user trial
57.6455026455026
70.15873015873
62.6772486772486
59.1375661375661
60.005291005291
61.2169312169312
62.4047619047619
59.1111111111111
55.7777777777778
Baseline Evalua-tion
54.0687134502924
69.7953216374268
62.2368421052631
58.2353801169591
60.9078947368421
59.9269005847953
60.4385964912281
56.8728070175438
55.3004385964912
5
15
25
35
45
55
65
75
85
95
SRL strategies profil
SRL strategy scales
%
t20=3.34, p=.003
Results: SRL
![Page 14: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/14.jpg)
• Scores are above centre line of the rating scale• Mentioned that prompts were not always presented at appropriate time
1
2
3
4
5
3.42 3.333.09
3.392.97 3.12 3.27 3.38 3.45 3.42 3.26
Learning Experience MSS
Items
Ra
tin
gs
Results: Learning Experience
![Page 15: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/15.jpg)
• Students were motivated• To perform simulation• Learn about interviews• To apply skills learnt in simulation in real world
• State motivation after simulation higher in 1st user trial compared to baseline
Baseline Evaluation First User Trial0
1
2
3
4
2.48
3.35
Motivation
Trial
Ra
tin
gs
t118.47=-8.64, p<.001
Results: Motivation
![Page 16: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/16.jpg)
• Comparison of expected and empirical MSS prompt distribution• More scaffolds in Information Management phase• Less scaffolds on reflection
Planning Informantion Management
Monitoring Debugging Reflection
Expected distri-bution
400.2 571.714285714286
400.2 301 343.028571428571
Empirical distri-bution
469 752 425 301 54
50
150
250
350
450
550
650
750
Expected vs. Empirical Distribution of MCSs
Phases
Fre
qu
enci
es
χ2(4,0.95)= 314.55, p<.001
Results: Scaffolding
![Page 17: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/17.jpg)
• Positional – time, date, simulator, etc.• “Rapport and Initial inquiry have been completed. Must now concentrate on moving
the interview forward”
• Technical – material addressing the system, questions or underlying model • Great concept. Has a lot of potential, will be much better when more points to consider
and points of information are added to the overview that accurately reflect performance.
• Notes – observations of the patient • “Px. unable to quantify length of period she has been like this for.
• Reflections – reflective text • it is quite important to deal with the patient with empathy to make them comfortable
and also to try and illicit the cause of the bout of depression, in this case. I felt I did not do too bad as far as expressing empathy is concerned, allowing the patient to open up and try and formulate a management plan. More experience in history taking will be the key
Methods: Coding reflective text
![Page 18: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/18.jpg)
• Corpus 1 (from cohort 1): had 358 unique text entries from 81 unique UserIDs giving an average of 4.41 entries per unique UserID• Engaged percentage = 57.9%
• Corpus 2 (from cohort 2): had 107 unique text entries into the note-taking tool from 50 unique UserIDs giving an average of 2.14 entries and 3418 views of the MST, resulting in 28 entries, 19 with text from 8 unique UserIDs• Engaged percentage = 35.7%
Results: Reflective Text
![Page 19: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/19.jpg)
• percentage breakdown of each type of content within the note-taking tool across the two cohorts:
Position Technical Notes Reflection
Cohort 1 15 33 25 69
Cohort 2 17 57 16 66
Results: Reflective Text
![Page 20: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/20.jpg)
• SRL• No correlation of log-data and SRL reports were observed – needs to be investigated
why? • No changes in SRL – long-term process
• MSS• Learners seem to need more assistance in effectively processing information by hints to
use more organizational, elaborative, summarizing or selective learning strategies• Are they more confident in the reflection phase and so pass the offer of scaffolds
• Time to engage• Less availability of simulators in cohort 2• Less lab time & shorter overall period
• Overall engagement and motivation increased• Higher percentage of time being scaffolded
Discussion
![Page 21: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/21.jpg)
• Provide MS at appropriate times
• Keep learners longer in simulation
• Provide additional services to promote SRL
Lessons Learnt
![Page 22: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/22.jpg)
What did we find out?
1. Is self-regulated learning supported?• Yes • Is it improved . . . . Maybe
2. Does the simulator augmentation through the service lead to better learning performance?
• No! But . . . . Yes!
3. Does the simulator augmentation through the service increase motivation?
• Yes
4. Is the service well integrated in the simulation and learning experience? • Yes
![Page 23: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/23.jpg)
• Longitudinal study• SRL changes slow• Integration of changes into behaviour
• Fuller analysis of competency• Metacognitive / SRL as well as that trained by the simulator• Surfacing of reflection / competency – social?
• Affect support• Encouragement / engagement / motivation• Please take our survey!!!
Further Work
![Page 24: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/24.jpg)
• The research leading to these results has received funding from the European Community's Seventh Framework Program (FP7/2007-2013) under grant agreement no 257831 (ImREAL project) and could not be realized without the close collaboration between all ImREAL partners.
Acknowledgements
![Page 25: An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators](https://reader036.vdocuments.us/reader036/viewer/2022062313/55842ad4d8b42a79568b4de1/html5/thumbnails/25.jpg)
http://bit.ly/smILEYCase sensitive!!!
@adam__moore
@ImREAL_project / www.imreal-project.eu
Thank-you!