The Science of Learning and theVirtual Anesthesia Machine:Benefits of "schematic" simulations in

learning about complex systems

Ira FischlerSimulation Faculty Learning Community

May 2008

Colaborators:

Sem Lampotang (Anesthesiology) Cynthia Kaschub (Psychology) David Lizdas (Anesthesiology)

And for jump-starting this effort: Sue Legg (Director, Partnership for Global Learning)

Plan for the talk:

“Learning with understanding:” The idea of mental models in psychology and education

How multimedia presentations can boost learning Potential advantages of simulation Transparent simulations and understanding The Virtual Anesthesia Machine (VAM) Learning with Transparent versus Opaque VAM Bridging abstract and concrete models: Mixed Reality

and the Augmented Anesthesia Machine A little bit about individual differences

The mini-science of learning What makes a difference?

Amount of practice (and the Power Law) Distribution of practice (and the Spacing Effect) Quality of practice (and Depth of Processing)

Making the information distinctive Building appropriate “mental models”

“I’m doing great in all my other classes. I read the book, came to class, outlined the material, and made flash cards, and still got a C.”

“Well, did you understand the material?”

“I thought I did…”

Mental models and schemas in comprehension

“If the balloons popped, the sound wouldn’t be able to carry since everything would be too far away from the correct floor. A closed window would also prevent the sound from carrying, since most buildings tend to be well insulated. Since the whole operation depends on the steady flow of electricity, a break in the middle of the wire would also cause problems. Of course, the fellow could shout, but the human voice is not loud enough for the sound to carry that far. An additional problem is that the string could break on the instrument. Then there would be no accompaniment to the message. It is clear that the best situation would involve less distance….

Mental models in cognitive science Term first used by Kenneth Craik (’43)

“If the organism carries a “small-scale model” of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilise the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to emergencies which face it.” (Craik, The Nature of Explanation, 1943)

Quality of the model depends on how well it captures the features of the domain that are critical for the task at hand

Understanding problems (Greeno, 1977) Our internal representation (or model) of the

problem should have accurate CORRESPONDENCE between relevant

elements in the world and model good COHERENCE between elements in the model appropriate links to PRIOR KNOWLEDGE that can

aid problem solving

environmentmental modelof problem

correspondenceco

he

ren

celinks to priorknowledge

Little things (can) mean a lot(aka the devil’s in the details) Subtle changes in problem “framing” can have

drastic effects on performance Effects of analogy on solving the “X-ray” problem

Preceded by bulb filament problem “fragile glass” framing: 33% then solve X-ray “laser intensity” framing: 69% then solve X-ray

Effects of lives lost/saved on risky decisions: Disease control programs, one more risky

“Lives saved” framing: 22% choose risky action “Lives lost” framing: 75% choose risky action

Came before text, historically Illustrations and drawings

To illuminate structure, function and relations

Animations and videos To make system dynamics visible

Interactive simulations To actively explore cause-effect dynamics, test

hypotheses, etc.

Advantages of “multimedia” can be dramatic:

Pictorial Representations

Mayer’s work on Multimedia (e.g., How lightning forms)

Compares.. text-only to text-with-illustration (often schematic) Narration-only with narration-plus-animation

Tests.. Retention by free recall of presented facts Transfer (understanding?) by generating solutions to:

Redesign (“how could you decrease lightning intensity?”) Troubleshooting (“how could there be clouds, but no lightning?”) Prediction (“what would happen with lower air temperature?”) Abstraction of Principles (“What causes lightning?”)

Retention and transfer with MM Retention: Modest MM gains

Across 6 studies, 23% gain, 0.67 effect size

Transfer: Dramatic MM gains Across 6 studies, 89% gain, 1.50 effect size

Mayer & Gallini, 1990: Lightning lesson

0

10

20

30

40

50

60

70

80

retention transfer

Perc

ent

Corr

ect

text with illustrations

text only

Potential advantages ofComputer-based simulation Cost: cheap systems, easy to replace, low risk Track performance and provide “just-in-time”

feedback on performance “Virtually Real” when needed But Reality can be played with:

Increase likelihood of rare but important events Increase salience of important features Present “hyper-real” depictions of space and time Make the abstract concrete, and the invisible visible

Instructional Choice-Points What do we want them to learn?

Declarative knowledge, procedural skills Immediate or long-term retention Reproductive or creative, flexible learning

How do we structure the learning? Amount of “grounding” in the domain Balance of guided (reception) and free (discovery)

learning Amount of online assessment and intelligent tutoring Student-tailored, or one-size-fits-all

Opaque versus Transparent Reality

Opaque representation: Simulation may be closely analogous to the physical system (iconic, concrete, high-fidelity, virtual reality) but hides underlying structure, functions and relations

Transparent representation: Simulation sacrifices physical fidelity but makes underlying aspects of system overt (abstract, idealized, schematic, symbolic)

Transparency in simulations Hollan’s STEAMER (1981) Goldstone’s Concreteness Fading (2004) Butcher’s simplified diagrams (2006) Debate focusses on “extent of fidelity” and

whether detail helps or hurts Little direct comparison

of simulation formats

The Opaque-Reality VAM

The Transparent-Reality VAM

The Virtual Anesthesia Machinewide use, little data

10 man-years of development time Available for free to individuals on the web Over 10,000 registered users Many positive reviews, both formal and informal Our goal: assess the effectiveness of VAM’s

Transparent Reality approach to simulation

Training Session

30-page instructional guide developed Provides foundation of knowledge

About anesthesia About the anesthesia machine and its subsystems

Guided tour of several subsystems Breathing circuit Mechanical ventilation Manual ventilation

Stresses visualization of dynamics using VAM

Workbook: Sample text Question 1: elimination of CO2. Are the gases exhaled

by a patient “scrubbed” of CO2 before entering the

bellows during mechanical ventilation?

Demonstration using VAM Simulation:

____ Click “Reset” to start simulation afresh

____ Point to the O2 flowmeter control knob to

enlarge it, then click-and-hold, and drag it counterclockwise until the O2 bobbin inside is

about halfway up the tube.

Workbook: sample text (cont’d)

What does this do? What happens to the flow of O2 from the supply line? Opening the valve increases the flow of O2 from the

supply line into the breathing circuit. Trace along its route through the plumbing. Where

does it wind up? It depends. For example, If mechanical ventilation is

selected, but not on, the O2 flows “backward” through the CO2 absorber, past the bellows and into the scavenger system

Performance on Day 2 Tests:undergraduate health majors

Performance on Day 2 Tests: 2nd-year medical students

Performance on AAA Board Exam Review questions (4AFC)

Judgments about VAM Confidence Judgments

On Component function: Significantly higher for Transparent VAM (p < .01)

On System dynamics: Marginally higher for Transparent VAM (p < .15)

Preferences for additional study 17 of 20 in Transparent group (UG) prefer TR VAM 11 of 20 in Opaque group prefer (UG) TR VAM 2 in TR, 7 in OR, think both would be preferable to either Similar trends among medical students; more want both

Where to next? Combination and order effects

Goldstone’s “concreteness fading” method?

More precise tests of transfer Transfer to procedural skill: does TR improve error

detection and response?

Hybrid simulations: John Quarles’ project

The Augmented Anesthesia Machine (AAM) Integrating transparent and realistic

representations with “mixed-reality” simulation

John Quarles and his Magic Lens

Declarative and Procedural Knowledge with VAM and AAM Two groups of undergrads Training:

Introduction to AM with VAM “positioning” components within the actual AM Five step-through exercises with VAM or AAM

Day 2 Testing Declarative: Board Exam Questions Procedural: Find a machine fault in the AM

Performance with VAM vs. AAM

Abstract and Concrete Knowledge

“Although the VAM may offer improved abstract knowledge, participants found it difficult to transfer this knowledge to the concrete anesthesia machine. This is precisely the concern that anesthesia educators have had with the VAM.

For example, many VAM participants understood the abstract concept of the inhalation valve and they correctly answered the written questions regarding the gas flow in the valve. However, during the fault test, they could not perform the mental mapping between the abstract representation of the VAM inhalation valve and the concrete representation of the real anesthesia machine inhalation valve. Thus, it was difficult for VAM participants to apply their abstract knowledge to a concrete problem, such as the problem presented in the fault test.”

Role of Spatial Abilities?

Three tests of spatial cognition Arrow-pointing working memory (small-scale) Perspective-taking (mid-scale) Navigating virtual environment (large-scale)

Correlations of spatial abilities and performance tend to be larger with VAM than AAM Suggests those with strong visualization skills can

compensate for impoverished materials

What we’ve learned Dynamic simulations can improve comprehension of,

and memory for, complex systems, BUT - Different kinds of simulation are optimum for different

kinds of learning So we need to know the goal of training

Experience with both abstract (schematic) and concrete (hi-fidelity) simulations may be optimum So we may need an integrated approach

Individual differences in domain-specific skills and abilities will impact effectiveness of representations But we need to know how much

It takes a Village (or at least a Learning Community) Cognitive/human factors psychologists Usability analysts Instructional psychologists and educators Simulation designers and engineers Domain experts and professionals

Thanks to all those RA’s

Emily McAlister Jonathan Greenwood Julianna Peters Shannon Bowie Sheila Holland Trudy Salmon