© 2012 aptima, inc. the science of game-based training effectiveness 29 march 2012 krista langkamer...
TRANSCRIPT
© 2012 Aptima, Inc.
The Science of Game-based Training Effectiveness
29 March 2012
Krista Langkamer RatwaniKara L. Orvis
© 2012 Aptima, Inc. 2
“It is fair to say that we currently ask young infantry [personnel] to function reliably across more domains than we ever have in the past, and the defense community has responded by inundating our forces with new technology and training to meet this challenge. Although this focus stems from the admirable desire to ensure that our forces receive the latest and greatest training and technology prior to combat, it has resulted in a patchwork of training efforts and technologies that have been implemented without demonstrating substantial proof of effectiveness.” - Marine Corps Gazette (2012)
Background
© 2012 Aptima, Inc. 3
Current State of Game-Based Training
Deployment of games making leaps and bounds– 2006: $125M spent on game-based training (Blunt, 2007)
Research making baby steps– Little evidence that games lead to learning (Blunt, 2007)
Research focused on whether serious games were motivating
– Training effectiveness studies are slowly leaking in but literature still disparate with mixed results (e.g. O’Neil and Perez, 2008)
Studying game-based training “in the wild” is hard Results are published across many domains and sources
– There continue to be many research questions regarding computer games and instruction (Tobias et al, 2011)
Little understanding of “why” serious games work
© 2012 Aptima, Inc. 4
Contributors to Training Effectiveness
Campbell & Kuncel, 2001; Colquitt, et al., 2000; Mathieu, et al., 1992; Salas & Cannon-Bowers, 2001
The Training Program
The Trainee
The Situational Context
© 2012 Aptima, Inc. 5
The Training Program: Things to Consider
What game features are most useful for targeting specific learning objectives? (Wilson et al., 2009)
Feedback– How often and what type of feedback should be provided?
Scaffolding– How can scaffolding approaches be incorporated into game-
based training?
Learning Objectives…
Should Drive…
Design Decisions!
© 2012 Aptima, Inc. 6
The Training Program – Empirical Evidence Example
Participants: 157 students playing a game that manipulated the level of Game Fantasy (low vs. high fantasy conditions)
Fantasy significantly impacted Declarative Knowledge and Application
F(1, 64) = 4.58, p = .04
F(1, 65) = 6.51, p = .013
Orvis et al., 2009
© 2012 Aptima, Inc. 7
The Trainee: Things to Consider
General training rule: Training should be difficult enough to increase motivation and performance, but not beyond the trainee’s capability– Attribute by treatment (ATI) interactions have been a focus within
the general training research (e.g., Gully et al., 2002)
Examples of Trainee Variables to Consider– Personality
Prior research in military settings has shown a positive relationship between personality variables and training effectiveness (e.g., Driskell et al., 1994)
– Prior videogame experience Research has shown that less than 50% of college age students play video
games on a regular basis (e.g. Orvis, Orvis, Belanich, and Mullin, 2005) Both general and genre specific game experience impacts how students
react to the game (e.g. Orvis, et al., 2005)
© 2012 Aptima, Inc. 8
The Trainee – Empirical Evidence Example
Participants: 139 students playing versions of the game G-DIS which manipulated task difficulty across 6 missions
1 2 3 4 5 61
2
3
4
5
6
7Adaptive Difficulty Condition
low openness
high openness
Mission
Per
form
ance
1 2 3 4 5 61234567
Static Difficulty Condition
low openness
high openness
MissionP
erfo
rman
ce
Bauer et al., in press
β14 = -.15, t(826) = -2.15, p = .032
© 2012 Aptima, Inc. 9
The Situational Context: Things to Consider
Training environment impacts effectiveness of training program (e.g., the game)
Organizational analysis is a critical component to effective training – even for serious games– Organizational or unit commitment to the training– Resources (e.g., time)– Involvement of the leader– Whether units are training together– KSA transfer environment– Available pre-training
© 2012 Aptima, Inc. 10
The Situational Context – Empirical Evidence Example
Participants: 165 Soldiers (representing 9 platoons) playing VBS2
Amount of Preparation prior to playing game and Amount of Leader Involvement during the game significantly impacted Unit Performance variables
F(1, 126) = 10.23, p < .01
2.5
3
3.5
4
3.19 3.30
3.90
Level of Leader Involvement During Training
Un
it P
roce
ss
F(2, 137) = 4.02, p < .05Ratwani et al., 2010
© 2012 Aptima, Inc. 11
The Importance of Evaluation
Evaluation is needed to actually assess learning Common methods of assessing training effectiveness
(Marine Corps Gazette, 2012)– Performance on the battlefield– Subjective assessments by trainees
Best practices– Longitudinal– Pre- and post-evaluation (assess skill level before and after
training) – Comparisons to other training methods (e.g., live training;
classroom training)– Good measurement
Include measures of the 3 factors just discussed Identify and develop standards of performance
© 2012 Aptima, Inc. 12
Continue to test the effectiveness of serious games
Understand how and why games lead to learning– Design characteristics– Learner characteristics– Context
Use scientific principles to build games that teach
Implications and Recommendations
© 2012 Aptima, Inc.