1 storytelling as instruction nov 06 anacapa sciences 7 november 2006 presentation to: storytelling...

1

Storytelling as Instruction Nov 06 Anacapa Sciences

7 November 2006

Presentation to: Storytelling as an Instructional Method Conference

Mesa, Arizona

Presentation by:Alan Spiker

Anacapa Sciences, Inc.Santa Barbara, CA

Scenario-Based Training: Lessons from Development and Implementation

2


Topics

What’s a Scenario and why is it needed?

Challenges Facing a Small Business Implementing SBT

Evolution of Scenarios as an Element of Training

Useful ISD Principles of Scenario Based Training

Measuring SBT Effectiveness

Lessons Learned: Navy, Commercial Aviation, Army

3


Topics (details)

What’s a Scenario and why is it needed for training? - look to the Thesaurus- many good reasons and some not-so good reasons for scenarios

Challenges Facing a Small Business in Implementing SBT- cajole not coerce, evince not convince, insinuate not dictate- implementing various types of SBIR-funded technologies

Evolution of Scenarios as an Element of Training- Geospecific DB --- > Framework ---- > Wizard ----- > Serious Games?

Useful ISD Principles of Scenario Based Training - eight that REALLY matter

Measuring SBT Effectiveness- Kirkpatrick’s 4-Level formal method- informal method in the real world

Lessons from Implementing SBIRs w/ Navy, Commercial Aviation, Army- Sometimes a great notion- Our scorecard: wins, losses, draws

4


What’s a Scenario?

1. An outline of the plot of a dramatic work, giving particulars to scenes, characters, situations, etc.

2. The outline or manuscript of a motion picture or television program, giving the action in the order in which it takes place, description of scenes and characters

3. An imagined or projected sequence of events, esp. any of several detailed plans or possibilities

4. Setting for a work of art or literature

5. Postulated sequence of possible events

6. An outline or synopsis of a play

Syn: scheme, plan, concept, sketch

Sources: Dictionary.com WordNet

5


What’s a Scenario really?

1. An outlineoutline of the plot of a dramatic work, giving particularsparticulars to scenes, characters, situations, etc.

2. The outline or manuscript of a motion picture or television program, giving the actionaction in the order in which it takes place, description of scenes and characterscharacters

3. An imaginedimagined or projected sequencesequence of eventsevents, esp. any of several detailed plans or possibilitiespossibilities

4.4. SettingSetting for a work of art or literature

5. Postulated sequence of possiblepossible events

6. An outline or synopsissynopsis of a play

7. Syn: scheme, plan, concept, sketch

Sources: Dictionary.com WordNet

6


Scenarios are good because they:

• Can be a script or a blueprint of what will happen

• Define roles of non-trainees

• Constrain allowable actions, decision points

• Help establish cognitive realism and trainee buy-in

• Lay out sequence of action to unfold

• They define which events occur, and when

• Give the storyline of actions

• Serve as communication conduit for training team

• And thus ……

7


Scenarios:

• Prevent chaos in the training environment

• Provide an antidote to unscripted free play in the sims

• Make training more interesting to the trainee

• Make training more believable – more face valid

• Make training more personal, customizable

• Are more consistent with our Experiential Learning Model approach to training adults

• Can’t substitute for measurable learning objectives

• Don’t ensure criterion-based training

• Don’t guarantee positive transfer of training

BUT Scenarios --

8


Our Challenge as a Small Business

• Anacapa Sciences – small business est. 1969• Implement SBT as part of SBIR products• Solicit claimants through insinuation, use of data-

driven approach, value-added consulting on the side

- so no organizational clout but that’s hardly unique to us

• Since 2000, attempted SBT in several SBIRs:– SamePage: team training in Shared Understanding for ARI (Ph. II)

– SimDATT: scenario-generation tool for NavAir (Ph. II+)

– (CT)2: web-based critical thinking training modules for ARI (Ph. III)

– TARGET: serious game based trainer (GBT) for ONR (Ph. I)

– Stalwart: Wiki-based team trainer for JFCOM (Ph. I)

9


Common Approach

• Identifying training need – why SBT needed in first place

• Finding suitable SBT sponsor – how SBT will help him/her

• Defining training baseline – what SBT is replacing

• Finding training objectives – usually a real scavenger hunt

• Selecting form of SBT – one size does not fit all

• Managing expectations – what SBT can and can’t do

• Measuring SBT effectiveness – how to know if it’s working

• Implementing SBT – what’s the exit strategy

• Assessing feasibility– who’s planning the birthday party

SBT is still training - Typical Steps in SBT Projects

10


Evolution of SBT Small Business Perspective

• Scenarios WEREWERE: Big, physical, fixed, geo-specific, 1-1 physical fidelity with criterion environment, created by SMEs, little measurement

• Scenarios ARE NOWARE NOW: Smaller, cognitive, dynamic, regional, 1-1 cognitive fidelity with criterion environment, created by project team, stopping points for measurement

• Scenarios WILL BECOMEWILL BECOME: Micro, meta-cognitive, adaptive, generic, 1-1 higher order fidelity with criterion environment, created by anyone, embedded measurement

11


Evolution of SBT Mission Scenario

trainee’s role

12


Evolution of SBT Instructor Script

instructor’sactions

13


Evolution of SBT Characteristic Features Now

• Synopsis – Storyline method of organization

• Trainee role specified through multiple means

• Timeline – used as scaffold on which to specify sequence of action

• Events – are specified, selected, and inserted as basic unit of action

14


Evolution of SBT Synopses - Storylines

Select scenario fromSelect scenario froma library of possiblea library of possiblesynopsessynopses

15


Evolution of SBT Trainee Roles

Conveyed through instructions, tables, special OPORDs,maps, immersion exercises, vignettes

(1) Evaluation of possible enemy courses of action that might disrupt our purpose in

occupying Fwd 1. The enemy’s primary methods of disruption are to promote fear and distrust – fear of them and distrust of US forces. Determine whether US occupation of Fwd 1 will diminish, enhance, or remain unaffected the enemy’s capacity to:

Conduct criminal actions against the general populace to induce fear in them. Conduct criminal actions against local leadership to induce fear in them. Conduct criminal actions against the general populace and local leadership to

demonstrate lack of capacity of US forces to protect them. Spread rumors that the US forces lack resolve to occupy Fwd 1 over a long period

of time.

Responsibilities on team

Estimate proactive and reactive enemy activities Predict the general enemy reaction to the positioning of our forces in Fwd 1. Build and interpret a link diagram that will describe the relations among and

between civil and religious leaders. Estimate the threat to logistics routes when asked by the Logistics Analyst.

Information/knowledge available to you in this mission 1. The Bn S-2 and the commander have pondered the nature of the enemy over the past

several months. What is listed on the OPORD, paragraph 1a is only a summary of their thoughts. Here is a more detailed general estimate:

Compare this photo to the OPORD map. Where is the train station in relation to Fwd 1? Who else should know about this? Why or why not?

16


Evolution of SBT Timelines – Action Flow

17


Evolution of SBT Event Selection and Specification

18


ISD Principles of SBTOverview

• Navair funded add-on to SimDATT project• Identified 69 principles from ISD realm that could be applied

to guide SBT development• User version incorporated into SimDATT to aid designer

Design Principle 1-4: Promote student motivation by instilling mental conflict.

Mental conflict can be provoked in students through several methods, including: 1) using facts that contradict their past experience, 2) involving a sense of mystery by presenting unresolved problems, and 3) providing scenario events that “baffle” their perceptions.

Explanation: Provoking mental conflict can stimulate a student's curiosity about the KSA he/she is about to learn and hence their motivation to learn. These “garden path” experiences, where students are essentially lured into an initial failure experience, help to form more accurate mental models and help them unlearn previously erroneous methods of looking at situa-tions and problems.

Example: There are a number of effective ways to trick a student to follow down a “garden path” in order to get their attention and impress upon them the difficulty of the to-be-learned KSA. Examples of five effective “garden path” tech-niques are shown in Figure 1-4.

Reference: Keller & Burkman (1993); Ross, Phillips, Klein, & Cohn (2005)

GARDEN PATH EXPERIENCE THAT PROMOTES MENTAL

CONFLICTTYPE OF KSA

Progressive increase in task difficulty until STUD’s skill level is exceeded

Prelude to learning to make precision approaches

Presentation of ambiguous stimulus that defies accurate ID

Prelude to learning VFR to IFR transition

Presentation of conflicting information that makes unambiguous decision difficult

Prelude to learning advanced aviation decision making skills (e.g., divert airfield)

Presentation of benign stimulus that is followed by abrupt malevolent change

Prelude to learning to recover from unusual attitude

Presentation of message that contains clauses which are logically or factually inconsistent

Prelude to learning ATC communication protocols

Figure 1-4. Five ways to promote mental conflict.

19


ISD Principles of SBTEight Really Useful Ones

ISD Design Principle

1-1 Use 5-part scenario structure

1-5 Link scenario objectives to student learning goals

2-1 Link scenario content to training objectives

2-8 Include meta-cognitive skills in scenario KSAs

3-2 Organize training by varying setup conditions

4-2 Use ‘telescoping’ approach to design scenario content

6-7 Use analogous cases as training scaffold

8-4 Stress critical thinking in debrief questions

20


ISD Principles of SBTFive-Part Scenario Structure

Storyline/Synopsis Players Events Content Timeline

Basic daytime flight from Dulles toNashville. During start, severalinterrupted start problems areencountered and discussed. NormalSID using autopilot. Practice areaentry for steep turns and stalls. Inpractice area, low hydraulic pressure(1) light comes on. After isolationand correction, descent is initiatedback to Dulles, using autopilot. Fullstop landing is completed.

ATC

ATIS

Dispatch

Interrupted Start

IFR Standard Departure

HYD 1 Low Pressure

Steep Turns

Stalls

SID Chart

CFM (company flightmanual)

Checklists

Start & Taxi — Interrupted Start

Departure– IFR

Enroute– Turns Stalls HYD

Approach– ILS

21


ISD Principles of SBTLink Objectives to Student Goals

The scenario will:

Expose you to different types of

Hydraulic Systems failures

Know bold-faced items by heart and engage correct emergency systems within

sec. of detection

After completing the scenario you will be able to

Subject you to different levels of low-vis conditions

Initiate VFR-IFR transition actions w/o hesitation

while remaining spatially oriented

Require you to practice multiple

approaches and make landings on runways

of various lengths

Internalize decision height actions and opt for

missed approach w/o hesitation

Expose you to high volume ATC traffic

under different amounts of radio

noise

“Tune” your attention to particular voices on the

radio and to assertively ask for a repeat transmission

when messages are inaudible

22


ISD Principles of SBTLink Content to Objectives

23


ISD Principles of SBTInclude Meta-Cognitive Skills

Generate mental simulations to troubleshoot emergenciesGenerate mental simulations to what-if a planGenerate mental simulations to ID route weaknessesEngage in counterfactual thinking (assume plan is wrong)Learn to size up a situationGenerate story to explain away inconvenient dataDiagnose cause of some flight disturbanceControlling attention through eye-scanning techniquesStress resistance trainingShared situational awareness

24


ISD Principles of SBTVary Setup Conditions

Variables that Promote Transfer of Training

High Level of Initial Learning

High Perceived Similarity between Training Task and Criterion Task

Tasks Learned in Relevant Context

Tasks Learned with Similar Cognitive Processes to Criterion Task

Example of Application in SBT

Variability in Training Conditions

Increase Difficulty of Instruction through Contextual Interference

Gradually Fade out Feedback and Knowledge of Results

Ensure that all core tasks receive many practice trials

Ensure that students are made aware of the task elements in training that will be present on the job (e.g., time pressure, equipment malfunctions)

Factors like time of day, amount of rest, time to prepare, etc. should be similar in training to that expected on the job

Ensure that tasks require a comparable level of critical thinking, decision making, problem solving, etc. as that expected on the job

Conduct training where environmental, aircraft, and flight conditions vary widely from session to session

Include contextual factors that make learning difficult, like frequent malfunctions, radio traffic, garbled comms, etc.

After initial sessions, don’t give student feedback after every takeoff/landing, but only after several

25


ISD Principles of SBTUse Telescoping Design to Create Content

(1)

(2)

(3)

(5)

(4)

26


ISD Principles of SBTUse Analogous Cases as a Scaffold

2 scenarios most likeyour present scenarioin their emphasis onCRM

27


ISD Principles of SBTStress Critical Thinking in the Debrief

28


Measuring SBT EffectivenessThe Formal Way

Evaluation Level Methodology

I User Acceptance Survey

II User Learning Comprehension Test

III Transfer to Job Gaining Unit Survey

IV Organizational Impact Longitudinal Study

Kirkpatrick’s 4-Level Approach to Evaluation

29


Measuring SBT EffectivenessReal World Metrics - Checklist

Does system have its own room?

Does system have a real-person as champion?

Does using community pony up some of its own money to keep system going?

Are they establishing training objectives for each scenario?

Do they call us (for tech support) to keep system going?

Do they return our phone calls and e-mails?

Do they invite us to their parties?

30


Lessons LearnedAnacapa’s Scorecard

• Why did SimDATT version for Navy fail?

• Why did SimDATT version for ASU-Mesa succeed?

• Why did SamePage come to a draw with the Army?

• Why might (CT)2 win/lose/draw with Army? - too close to call now

Our Record is 1-1-1-1Our Record is 1-1-1-1

31


Why SimDATT Failed with Navy

• Users liked the aviator interface and content• But didn’t need scenarios for their part-task training• Either FRS or refresher• So all the marks appear high in Level I-II evaluation• But our real-world metrics are very poor

32


Why SimDATT Succeeded at ASU-Mesa

• Same basic system as for the Navy• ASU-Mesa embraced scenarios for their students• Make even initial training LOFT-like• Have intimate relationship with the “gaining unit”• Supporting a four-level Kirkpatrick evaluation

33


Why SamePage Stalled with Army

• Users liked the concept• Users liked the immersion experience• Users liked the challenge, great for leader development• But need domain changes to cover multiple MOSs• Needed a wizard, not a framework

34


What We Needed Was Something Like this

35


And This

36


Why (CT)2 Still in Limbo with Army

Have conducted multiple evaluations, all very positive

Contains multitude of cognitive vignettes

But where to put into curriculum?

What has to drop out to put (CT)2 in?

Yet our inputs reached other parts of CGSC curriculum

37


Trends in SBTWrap-up

• SBT becoming more adaptive• SBT could become more meta-cognitively oriented• SBT will become more SCORMED• SBT scenarios will be updated often, perhaps daily• SBT will become more personalized (e.g., wikis,

avatars)• SBT will become “jazzed up” to support a market-

based approach to training (how many hits did you get today Mr. Training Manager?)

• SBT will become SBD, scenario based development

38


Meta-Cognitive Skills in ScenariosEffects Based Thinking Tool

39


Meta-Cognitive Skills in ScenariosUse of Wikis

Key Features• Search• Edit• Save entries• Color code edit versions• url is:

http://www.anacapatechnical.com/projects/jtf

1 storytelling as instruction nov 06 anacapa sciences 7 november 2006 presentation to: storytelling...

Documents

anacapa sciences scenarios

anacapa sciences whats

anacapa sciences topics

action characters

sequence of action

ca scenariobased training

alan spiker anacapa

army slide