serious gaming: research
TRANSCRIPT
1Challenge the future
Serious Gaming Research© Dr. Igor Mayer [email protected]
Faculty Technology, Policy and Management (TPM) & Signature Games
Delft University of technology, The Netherlands
2Challenge the future
3Challenge the future
References (1)1. Mayer, I. S., Bekebrede, G., Harteveld, C., Warmelink, H. J. G., Zhou, Q., van Ruijven, T., … Wenzler, I. (2013). The
research and evaluation of serious games: Toward a comprehensive methodology. British Journal of Educational
Technology, n/a–n/a. doi:10.1111/bjet.12067
2. Mayer, I. S., Bekebrede, G., Warmelink, H. J. G., & Zhou, Q. (2013). A Brief Methodology for Researching and Evaluating
Serious Games and Game-Based Learning. In T. M. Connolly, L. Boyle, T. Hainey, G. Baxter, & P. Moreno-Ger (Eds.),
Psychology, Pedagogy and Assessment in Serious Games (in press) (pp. 357–393). IGI Global. doi:10.4018/978-1-4666-
4773-2.ch017
3. Mayer, I. S., Kortmann, R., Wenzler, I., Wetters, Á., & Johan, S. (2014). Game-based Entrepreneurship Education:
Identifying Enterprising Personality, Motivation and Intentions amongst Engineering Students. International Journal of
Entrepreneurship Education (in press).
4Challenge the future
References (2)
1. Mayer, I. S., van Dierendonck, D., van Ruijven, T., & Wenzler, I. (2013). Stealth Assessment of Teams in a Digital Game
Environment. In GALA 2013 Conference, Paris (pp. 1–13). Paris, France: Springer.
2. Mayer, I. S., Warmelink, H. J. G., & Bekebrede, G. (2013). Learning in a game-based virtual environment: a comparative
evaluation in higher education. European Journal of Engineering Education, 38(1), 85–106.
doi:10.1080/03043797.2012.742872
3. Mayer, I. S., Warmelink, H. J. G., & Zhou, Q. (2014). The Utility of Games for Society, Business and Politics: A Frame
Reflective Analysis. In Nick Rushby & D. Surry (Eds.), Wiley Handbook of Learning Technology (in press). Wiley.
4. Mayer, I. S., Wolff, A., & Wenzler, I. (2013). Learning Efficacy of the “Hazard Recognition” Serious Game: A Quasi
Experimental Study. In M. Ma, M. F. Oliveira, S. Petersen, & J. Baalsrud Hauge (Eds.), 4th International Conference, SGDA
2013, Trondheim, Norway, September 25-27, 2013. Proceedings (pp. 118–129). Berlin Heidelberg: Springer Verlag.
doi:10.1007/978-3-642-40790-1_12
5Challenge the future
Review articles
1. Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical
education: A literature review. Computers & Education, 53(3), 603–622. doi:10.1016/j.compedu.2009.04.001
2. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of
empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686.
doi:10.1016/j.compedu.2012.03.004
3. Vogel, J. J., Vogel, D. S., Cannon-Bowers, J. A., Bowers, C., Muse, K., & Wright, M. (2006). Computer Gaming
and Interactive Simulations for Learning: a Meta-Analysis. Journal of Educational Computing Research, 34(3),
229–243. doi:10.2190/FLHV-K4WA-WPVQ-H0YM
4. Gosen, J., & Washbush, J. (2004). A Review of Scholarship on Assessing Experiential Learning Effectiveness.
Simulation & Gaming, 35(2), 270–293. doi:10.1177/1046878104263544
6Challenge the future
Towards a Science of SG (SoSG)?
Frames and discourses Methodology Research designs
and data-gathering
Validated research instruments and tools
A dynamic body of knowledge identifying
the state of the art and knowledge gaps.
Professional ethics of the SG designer, the SG advocate, the SG
seller, the SG interventionist, etc.
7Challenge the future
Game theory versus gaming
Game theory Gaming
Rigid rule-based, closed Free form, seminar, open
Formal, mathematical, quantifiable, economics,
psychology
Informal, social, interpretative and qualitative,
social, political intervention sciences
Experimental control, objective, separation
researcher and subjects, large ‘ n’ , player cannot
change the rules or setting
Semi- or non experimental, subjective, interaction
between researcher and subjects, small ‘ n’ ;
players should change the rules or setting
Theory-based, hypothesis testing, generalization,
prediction and forecasting
Theory construction, exploratory, hypothesis
formulation, constructing the future
‘ Players’ are research objects;‘Players’ are learning subjects; they bring with
them tacit knowledge, social relations etc.
8Challenge the future
Framing game related research
Frame Example
Research theory Game theory as in economics, political science
Research concept Political decision-making as a strategic game
Research object Studying game cultures, game economics, game
politics
Design artifact Socio-technical design etc.
Research method Quasi-experiment
Intervention method Therapy, learning, change or decision-method
Data-gathering method Observation, group interview, data-modeling
9Challenge the future
Strong/weak aspects of gaming as a research method
Strong aspects Weak aspects
Flexible, adaptable, multi-purpose,
complementary
Time and resource consuming. Difficult to
manage. No guarantee for success.
Multi-, interdisciplinary, innovative, appealing Problematic legitimization as ‘science’
Research in action, interaction with real
stakeholder, problem owners, users, etc.
Dependency on others (financers, players,
stakeholders, designers).
Deliverable by or end product. Game takes over the research
Concrete, practical, tangible, fun A lot of fuzz that is not research
… …
10Challenge the future
Requirements for SG Meth.
Requirements Meth. SG Research
Broad in scope
Comparative
Standardized
Specific
Flexible
Triangulated
Multi-leveled
Validated
Expandable
Unobtrusive
Fast and non-time
consuming
Multi-purpose
d
11Challenge the future
Underlying research questions
•(How) do (virtual) gaming experiences (a)(e)ffect learning of professionals in socio-technical systems / multi-actor contexts?
•(How) does (virtual) game-based learning (a)(e)ffect Real World (RW) policy-making in socio-technical systems / multi-actor contexts?
What is the relation between (virtual) gaming and (professional)
learning in socio-technical systems / multi-actor contexts?
•Development and testing of design and validation theories, methods & tools•Validation studies of specific and generic game-based artifacts and events.
Design-oriented research (artifact):
‘making it (better)’
•Development and testing of game evaluation, measurement and feedback theories, methods & tools
•Studies into the learning effectiveness of game-based interventions•Studies into the transfer game-based interventions to the RW
Policy, management oriented research (intervention):
‘making it work’
•Studies into the design and use of SG as a research instrument, e.g. e.g. quasi experimental, questionnaires, video observation, Q-method, etc.
•Studies using game based research in infrastructures.
Domain-oriented research (energy, rail, water, ports, tunnels etc.):
‘making it matter’
•Studies on SG in a cultural, organizational, political, economic context•Theory construction on serious gaming ‘as’ / ‘in’ socio-technical / multi-actor systems
Scientific game research:
‘making it understandable’
12Challenge the future
Serious gaming
Case study
Computer simulation
Experimental research
Participatory research methods
Interviews
Surveys
Document
analysisGroup interviews focus groups
Lage schale data
analysis
Scenario analysis
Panel research
Quasi experim
ental research
Bio & psycho-Metric tests
Q methodo
logy
Videoobservat
ion
(participatory)
Observation
Eye tracking
Crowd sourcing
Path tracking
Mayer, I. S., Warmelink, H., & Bekebrede, G. (n.d.). Learning in a Game-based Virtual Environment: a Comparative Evaluation in Higher Education. European Journal of Engineering Education.
13Challenge the future
Shortcomings
Few indications how to use the models, for what purpose, with what scope and under what conditions.
Few procedures how to validate the conceptual research / evaluation model.
Few research hypothesis and research designs.
Few definitions, relations and interrelations between the concepts in the model.
Few operationalization and validation of constructs.
Furthermore in the application of the models we see:
A dominance of single case studies, one game, one context of application.
Lacking information on the questionnaires used.
A focus on GBL of children in formal education; little attention to advanced-professional learning, outside education;
A focus on learning of individuals in formal training or educational context; little attention to learning of teams, groups, organizations, networks or systems in policy or organizational context;
14Challenge the future
Research framework
2. Theory: Complex, multi actor systems and policy
making.
3. Method: Modeling, simulation & gaming
(MSG)
1. Domain application: infrastructures
1.1 Water, Rail, Ports, Energy, Tunnels, etc...
FramingFraming
Sub question 4
sub
que
stio
n 6
Research question 2
Research question 1
Research question 3
Sub question 5
2.1 Complex adaptive systems, resilience, integrated planning,
self-organization, sense-making, etc...
3.1 Design and evaluation of (Serious)
Gaming-Simulation
Fram
ing
Case-Experiments,
e.g. Levee Patroller, SimPort, Water Game,
15Challenge the future
Research designMSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP
When? Pre-game In-game Post-game
Observation number
O1
O2 O3 O4 O5 O6
How? Online surveyPaper
quest.Paper
quest.Paper
quest.End of game debriefing
Online survey
What? Soc. Dem. Analysis of
maps
Involvement in
MSPMSP
processMSP
processInfluence
After action review
Knowledge in
MSP
Influence in MSP Game play
MSP in country Emotions
Valid response 63 50 40 41 41 38
Additional data gathering
Video registration – Observation – Data logging
16Challenge the future
Quasi experimental design(Mayer, et al. under review EJEE, BJET, 2012)
XSerious Gaming session
O3
Time
In game data logging
Obse
rvati
on
O4
Learning proces
O2
Inte
rventi
on p
roce
ss
Tra
nsfe
r pro
cess
O1 = Observation 1 Learning objectives, contextO1 = Observation 2 pre-game measurementX1 = Intervention = gaming sessionO3 = Observation 3 post game measurementOx1...n = In game observations / measurementsO4 = Observation 4 Learning objectives, context
O1 Case study
Ox1 Ox3Ox2 Oxn
17Challenge the future
Measuring indicatorsMSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP
Quantitative Qualitative
Self-reported by the participants
The level of engagement of the players
in the game.
The influence attributed to each
stakeholder, by other stakeholders.
The quality of each of the four MSPs as
assessed by each stakeholder on
ten criteria.
The lessons and insights on the
process and outcome reported
during and after the game.
Observed by the facilitators
The square nautical miles assigned to
the different spatial functions in the
four MSPs.
The observations on how the MSP
process goes, by the game
facilitators.
The quality of each of the four MSPs.
The overlap, conflicts, internal
contradictions etc. between
different spatial functions in or
between MSPs.
18Challenge the future
• References
• Mayer, I. S., Warmelink, H., & Bekebrede,
G. (n.d.). Learning in a Game-based Virtual
Environment: a Comparative Evaluation in
Higher Education. European Journal of
Engineering Education.
• Mayer, I. S. (2012). Towards a
Comprehensive Methodology for the
Research and Evaluation of Serious
Games. Vs games (pp. 1–15). Genoa:
Procedia Computer Science 00 (2012) 000–
000. Retrieved from
www.sciencedirect.com
19Challenge the future
EXPKNOW_PRE
SOCSKILLS_ PRE
PROFSKILLS_ PRE
Learning expectations
AGE
SEX
ACHIEVEMENT
Learning style
preferences LRNPREF
Attitude VLEVALUEDL
Attitude GBL EXPSERGAME_ PRE
Engagement FUN
- .12
.2 (t-test)
.12*
.18
.2
.42
0.11
.24
.17
.25
.28*
.40
.46
.14
.42
FREQ. DIGITAL GAMES
FREQ NON-DIGITAL GAMES
FREQ SG
1.1 (t-test)
Attitude GBL EXPSERGAME_POST
EXPKNOW_POST
SOCSKILLS_POST
PROFSKILLS_ POST
Learning satisfaction
MOTIVATION
.22
.22
.19
.34
.24
.1
.24
.13
.16*
.17*
.18*
.17*
.27
.26
.39
.27
.39
-.17
.16
.12*
.3
.1
.62
.34
.27
.18
.32
.46
.53
.58
.78
Quality VLE CYBFUNCTION;
CYNCLEARNESS; CYBACCESS; CYBIMPRESS; CYBCONTENT
Quality game design
QUAGAME; RELGAME
Quality facilitator QUAFACIL.
Game play QUAPLAY;
EFFORTPLAY
Q game
Would like to play other SGs in education.
.2*
.45
.36
.41
.43
.26
.28
.51
.38
.44
.39
.38
.34
.48
Structural equation modelling (Mayer, et al. under review EJEE, 2012)