Download - A Research Approach to Develop Measurable Competencies and Skills: Videogames or Real-World Training
A Research Approach to Develop Measurable Competencies and Skills:
Videogames or Real-World Training
K. Anders Ericsson
Department of PsychologyFlorida State University
by
One of the crucial issues of traininginterventions is to be able to specify their goals so one can measure andevaluate objective outcomes.
I believe that videogames and simulatorscan provide us with the tools to specifythe goal and targets of educationfor everyday and professional life.
Getting away from paper-and-pencil or multiple-choice tests of the type of Scholastic Aptitude Tests (SAT)
Define outcomes with video games and simulations of everyday situations and tasks
TraditionalInstruction
On-the-jobexperience Target
Performance
“Retain skills and knowledge”“learning transfer”
“generalization”
TraditionalInstruction
On-the-jobexperience Target
Performance
“Retain skills and knowledge”“learning transfer”
“generalization”Alternatives,such as
SimulationsVideogames
Outline of My PresentationI. If the desired competency concerned a domain,
like Chess, Music, or Sports, how would expert performance be attained
II. Objective measurement of performance
III. Mechanisms, skills, and their acquisition through deliberate practice
IV. Acquisition of control and complex mental representations— Beyond automaticity V. Training and the fidelity assumption
I. If the desired competency concerned a domain, like Chess, How is Expert Performance Attained
How did chess players attain their performance?
Benjamin Franklinlearned to play chess from an acquaintancein 1733 He was one of the firstchess players in America. Although he was an avidplayer he reached a modestlevel of chess skill
Our Knowledge about Learning and Skilled Performance is Limited
How good is a chess player or an individual in a domain of expertise?
--Self ratings
Professors Drivers Doctors(94% above average) (80% above average) (Davies et al., 2006)
Self-Assessments on Relative PerformanceHow would you rate your own performance?bottom 10% below average average above average top 10%
Professors Drivers Doctors(94% above average) (80% above average) (Davies et al., 2006)
Self-Assessments on Relative PerformanceHow would you rate your own performance?bottom 10% below average average above average top 10%
How could a chess player in the chess club be able to keep improving?
Playing games
Watching better players play
Reading books
Improving One’s Chess Performance orPerformance in Tennis, Golf or SCRABBLE
Rated Relevance for Improvement from Doing Xon a scale from 1 to 7 (Charness, Krampe, & Mayr, 1996)
Active participation Noin chess tournaments 6.1 correlation
Playing chess games Negativeoutside of chess 3.6 correlationtournaments
Serious analysis of Positivepositions alone 5.9 correlation
Rated Relevance for Improvement from Doing Xon a scale from 1 to 7 (Charness, Krampe, & Mayr, 1996)
Active participation Noin chess tournaments 6.1 correlation
Playing chess games Negativeoutside of chess 3.6 correlationtournaments
Serious analysis of Positivepositions alone 5.9 correlation
Effects and Dose-Response Relations to Practice Activities
Professional teachers and coaches
* Monitor students’ development* design training tasks for individual students
Expert Performance
Rated Relevance for Improvement from Doing Xon a scale from 1 to 7 (Charness, Krampe, & Mayr, 1996)
Active participation Noin chess tournaments 6.1 correlation
Playing chess games Negativeoutside of chess 3.6 correlationtournaments
Serious analysis of Positivepositions alone 5.9 correlation
Simulated Play Against World Class PlayersStudy published games by chess masters
Make predictions for each next moveCheck if your prediction was correct, if not, study the chess
position until you understand why the correct move was played
Black on move
Position B
On-the-job supervised experience
Classroom Knowledge Acquisition Measurable
Outcomes
Part II. Objective Measurement of Performance
Independent ability to increase patients’ health
Kirkpatrick’sFour LevelModel ofTraining
Evaluation
The most respected experts (“expensive” stockbrokers) are not markedly better in picking stocks on New York Stock Exchangethan a random process, such as a monkey throwing darts (c.f. Wall Street Journal study)
Superior PerformanceChess
winning chess games
50 games (100-250 hours) for a rating
Psychotherapysuperior ability to increase
mental patients’ health (post-pre)50-100 patients (400-800 hours)
Teachingsuperior ability to increase
students’ performance (post - pre)
500-1000 contact hours
MinimalEffects of
Traditional Indicators
of Expertise
Long Education
Extended experienceas a professional
Teaching
Psychotherapy
Identify challenging and difficult situations, where experts are supposed to excel.
Adriaan de Groot (1914–2006)
Recreate the situation
and taskin laboratory
Actions and thoughts
of novicesand experts can
be directly compared
Select the best chessmove for this position
Type as much of thepresented text as possiblewithin one minute
Play the same piece ofmusic twice in samemanner
Chess
Typing
Music
Domain Presented Information Task
Selection
Speed
Control
Reproducible Soccer Performance
From cover story in Time Magazine (March 10, 2008) on The Science of Experience
Response to Critical Unexpected Problems
Salchow
Part III. Mechanisms, Skills, and their Acquisition through Deliberate Practice
The problem with learning during work and play
Andy Murray (Scotland) in 2010
The problem with learning during work and play
Immediacy of Informative Feedback
Sport training performance(1-2 years)
Investing value (10-20 years)
Medicine patient outcomes (12-36 months)
Increase in Complexity and Control as a Function of
Years of Piano Training
Yearsof
pianotraining
“individualized training activitiesespecially designed by a coach or teacher to improve specific aspects of an individual's performance through repetition and successive refinement.
To receive maximal benefit from feedback, individuals have to monitor their training with full concentration, which is effortful and limits the duration of daily training”.
(Ericson & Lehmann, 1996, pp. 278-279)
Deliberate Practice
Design and Sequencing of Training Activities
Professional teachers and coaches
* Monitor students’ development* design training tasks for individual students
Expert Performance
Accumulated Amount of Practice Alone During the Development
of Amateurs and Expert Musicians
The Body’s Resilience to Change The Protection of Equilibrium
Increase of Fitness
DeliberatePractice
Optimal
How many consecutive push-ups can someone perform without pause?
(H. H. Hart, 1974)
Range of modifiability with extendedDeliberate Practice
Charles Linster
Non-stop push-up records
http://www.recordholders.org/en/list/ulysses.html
6,006 Charles Linster (USA) 05-Oct-1965
…
10,507 Minoru Yoshida (JAP) Oct-1980
Fu Bingli, a kung fu master from China 12 press ups with right fingerhttp://www.telegraph.co.uk/news/newstopics/howaboutthat/6385592/Man-proves-he-has-worlds-strongest-fingers.html#Accessed on October 23rd, 2009
Part IV. Acquisition of Control and Complex Mental Representations—Beyond Automaticity
The Development of Increased ControlActively Avoiding Automating Control of Critical Aspects of Performance
Reproducibly Superior Performance is not automatic but is associated
with informative verbalizations of thoughts
Black on move
Position B* What aspect was overlooked* When could this aspect have been
discovered* How to avoid similar mistakes in the future* Develop new skills by deliberate
practice
Learning from Making an Incorrect Move
Alekhine beat most of the 30 skilled players while playing them simultaneously under blindfold conditions
Ref
P16A
http://images.google.com/imgres?imgurl=http://www.chessbase.com/news/2007/bilbao04.jpg&imgrefurl=http://www.chessbase.com/newsdetail.asp%3Fnewsid%3D4189&h=318&w=480&sz=36&hl=en&start=1&usg=__JwOiqXRmMEJcHCssYGF1N_5BqU4=&tbnid=DC9VzEbhAUofSM:&tbnh=85&tbnw=129&prev=/images%3Fq%3Dblindfold%2Bchess%26gbv%3D2%26hl%3Den%26sa%3DGAccessed on August 26, 2008
Blindfold Chess
Exceptional abilities
The Importance of Acquired Mental Representations
Expert Performers are Better Able to Represent and Analyze Situations
Slowing Down(Moultonet al.,2010)
(Ward &Williams, 2003)
(Tuffiash, Roring, & Ericsson, 2007)
(Kesting, et al., 2010)
V. Training and the Fidelity Assumption
It is frequently assumed that the best learning of skills occurs in the natural environment .
Hence, simulation should therefore reproduce the experience in the natural environment
What do we know about natural learning?How optimal is that type of natural learning?
Perceptual Performance on Cardiac Auscultation
Instruction and Experience
0-9 years 10-20 years
Over 20 years
General Practitioners
Student
Based on Butterworth & Reppert (1960)
Reviews (Choudhry, Fletcher, & Soumerai, 2005; Ericsson, 2004; Ericsson, Whyte, & Ward, 2007)
Perform
ance
No Correlations between Amount of Experience and Performance after the First Year
Benefits of Provided Experienceon Performance
Use of Cockpits with Visual and Kinesthetic Simulation
Modest transfer of performance—savings of training in real airplanes (except new procedures)
(Allerton, 2000; Roessingh, 2005; Rantanen & Talleur, 2005)
Focus on Fidelity and SafetyA airline captain needs 3,500 hours of flying plus 60 hours of simulation
(Parker, Johns, & Hellige, 2007)
Identifying the Crucial Elements of Effective Simulator Training - Deliberate Practice
A review of 109 studies of high-fidelity medical simulation showed thatthe characteristics of Deliberate Practice accounted for improvements
(Issenberg et al., 2005)
A review of 31 of those studies that examined repetitive practice, and “found a strong association (η2 = .46) between hours of practice on high-fidelity medical simulators and standardized learning outcomes.” (Issenberg et al., 2006, p. 792)
The Use of Simulators for Deliberate Practicewith Performers at Different Levels of Expertise
Allows exposure to challenging situations without danger
Allows exposure to difficult situations under conditionsoptimal to learning and performance - Individuals being focused and ready - Immediate feedback (and opportunity for repetition) - Presentation of related cases to facilitate discrimination
Allows the presentation of rare emergency situations
Deliberate Practicewith Simulators
Authentic test conditions with actors(Kneebone et al., 2005)
Reinstating actual scenariosin anesthesiology (Liu et al, 2009)
Measurement and TrainingCapturing the essence of expert performanceMeasuring it with representative tasks
Measuring Identification
current level of of optimal
performance training tasks
TraditionalInstruction
On-the-jobexperience Target
Performance
“Retain skills and knowledge”“learning transfer”
“generalization”Alternatives,such as
SimulationsVideogames
RecommendationsIdentify skills and analyze domains of activity
where performance can be measured with objective methods that capture on-the-job performance
with large individual differences in attained performance where increases in performance motivate major investment
Develop libraries of representative situations with appropriate feed-back about correct/appropriate actionswith scaled difficulty to maximize optimal training effects
Develop cognitive structures to support Deliberate Practiceassessment of representations for thinking, planning, and
evaluations in Long-term working-memory (LTWM)essential for “Retain[ing] skills and knowledge”
“learning transfer” “generalization”
The Complex Process of
Acquisition ofof Expert
Performance