Gamification Research: What the Numbers Reveal

By: Karl M. Kapp, Ed.D.

Professor, Instructional Technology Bloomsburg University

Twitter: @kkapp

Lynda.com Course: Gamification of Learning YouTube Video

Web Site: www.karlkapp.com

Books

Getting in touch with Karl

Twitter: @kkapp Web: www.karlkapp.com

Email: karlkapp.com : http://www.facebook.com/gamificationLI

Co-Researcher • Deltcho Valtchanov

– An experimental psychologist at the University of Waterloo Games Institute. His research focuses on how real and virtual environments can influence cognitive function, and how games and gamification can be used to motivate positive changes in behaviors."

Agenda • What did we study?

– Question Driving Research

• How Research Was Conducted – Procedure/Methodology – Axonify Platform

• Research Questions • Results • Discussion

Acknowledgement

• Research was graciously funded through a grant by:

What are we discussing?

• Results of a research study: – Submitted for peer-review – Conducted to academic standards – Not a case study

• Applied Research/Field Study – Not experimental – No control group – Not exactly the same population

Question driving research

Is it possible that a casual game can be used as a “hook” to motivate a learner to engage with content that is instructional but not part of the game play.

In other words…

Does a casual game encourage learners to engage with a gamified learning platform more than asking them questions?

Literature Review

• What do we know? – Several theories support the idea of serious games

positively influencing motivation (Garris et al. 2002; Ryan & Deci 2000; Ryan, Rigby, & Przybylski 2006 ).

– Wouters et al (2013), found in their metanalysis of 39 game studies that serious games are not more motivating than other instructional methods used in comparison groups.

Literature Review

• What do we know? – 55% of teachers use games in the classroom on a

weekly basis (“Teachers Surveyed” 2013) – 20% of corporations use games for learning

(“Association for Talent Development” 2014) – Marketplace of learning games or “serious games” is

predicted grow at a compound annual growth rate of 16% reaching an estimated value of $5,448 million by 2020 (“Serious game” 2015).

Literature Review

• What do we know? – 55% of teachers use games in the classroom on a

weekly basis (“Teachers Surveyed” 2013) – 20% of corporations use games for learning

(“Association for Talent Development” 2014) – Marketplace of learning games or “serious games” is

predicted grow at a compound annual growth rate of 16% reaching an estimated value of $5,448 million by 2020 (“Serious game” 2015).

Literature Review

• What do we know? – Only a third of U.S. employees are engaged at work – 50.3% report themselves as "not engaged" at work – 16.8% reported themselves as "actively disengaged"

(Adkins 2016).

How was research conducted?

• Two Groups/Conditions – Game Group – No Game Group

• Length of Study – 12 month period of study

• Data Collection – In place data set – Data mined from existing database

• Operationally Defined – Casual Games – Learning – Engagement

Participants

• Game Group – Retail Sales Company with 1,908 employees in a that

used casual games as motivation for learning • No Game Group

– Retail Sales Company with 4,393 employees in a retail chain that did not use casual games to motivate learning.

• Note: Unequal distribution of employees between the two major retail chains was a result of one retail chain being much larger than the other.

Data Collection Axonify Platform

• Software platform was used by multiple clients who both used casual games to engage learners and those who did not.

• System collected and stored a variety of data related to learner interactions on the platform which made it possible to track and examine the data set anonymously.

• Game and no-game condition identical in every way except for the one variable of game or no-game.

• Funded research.

Operational Definitions

• Learning (Recall) – “The more one knows (remembers), the more

intellectual competencies one has to draw on for thinking, problem solving and even creativity…knowledge and skills are acquired through memory” (Klemm 2007, p. 63).

– “the ability to remember an idea, material or phenomenon in a form very close to that in which it was originally encountered” (After Taxonomy 1956; Seels & Glasgow 1990).

Operational Definitions

• Casual Game – “In general, casual games involve less complicated

game controls and less complexity in terms of gameplay” (Loreto & Gouaïch 2010).

• Low Barrier to Entry • Played in Short Increments (5-20 minutes) • Reduced Complexity • Non-punishing game play

– Commercially popular casual games include Angry Birds, Bejeweled, Diner Dash, Candy Crush and Microsoft’s Solitaire.

Research Question One

• Does the opportunity to play a game impact level of employee engagement?

• For the purpose of this study, level of engagement was operationally defined as: – Number of times a learner returns to the software. – Number of sessions a learner spent browsing other

related parts of the platform (e.g., Report Card) and voluntarily chooses to participate in extra learning opportunities.

Research Question Two

• Does a higher level of engagement correlate with increased learning?

• For the purpose of this study, level of learning was operationally defined as: – Number of correct answers provided by learners over

a 12 month period. – Correct answer streak length

• Remember: learning is defined as: “the ability to remember an idea, material or phenomenon in a form very close to that in which it was originally encountered”

Null Hypothesis

• There will be no difference in the level of engagement between the two groups.

• Level of learning of employees allowed to play a game prior to being presented with content to be learned will not be different than the level of learning by a group of employees not allowed to play a game prior to being presenting with content to be learned.

Research Question One: Results

• Does the opportunity to play a game impact level of employee engagement? – Using a one-way ANOVA (Analysis of Variance of the Means) – Learners in the game condition logged in to do their training

significantly more often (M = 108.12 per year) than those in the no-game condition (M = 71.30) p < 0.001.

– Learners were significantly (51.64%) more motivated to come back and engage with the learning platform when they could play a game.

Statistically highly significant is often stated as P < 0.001 (less

than one in a thousand chance of being wrong).

Research Question One: Results

• Does the opportunity to play a game impact level of employee engagement? – Using a one-way ANOVA (Analysis of Variance of the Means) – Learners in the game condition logged in to do their training

significantly more often (M = 108.12 per year) than those in the no-game condition (M = 71.30) p < 0.001.

– Learners were significantly (51.64%) more motivated to come back and engage with the learning platform when they could play a game.

Research Question One: Results

0

20

40

60

80

100

120

Games OFF Games ON

Aver

age

Num

ber o

f Log

ins i

n 12

Mon

ths

Condition

p < 0.001 108.12

71.30

Research Question One: Results

• Does the opportunity to play a game impact level of employee engagement? – A one-way ANOVA revealed that learners in the games condition

viewed their report card significantly more often (M = 9.88) than learners in the no-games condition (M = 2.96) p < 0.001

– The analysis also revealed that learners in the games condition volunteered to do significantly more extra training material (M = 6.28) than learners in the no-games condition (M = 3.3) p < 0.001

Research Question Two: Results

• Does a higher level of engagement correlate with increased learning? – Using a one-way ANOVA the analysis found that learners in the

game condition answered significantly more questions correctly (M = 413.47) than learners in the no-game condition (M = 145.61) p < 0.001.

Research Question Two: Results

• Pearson correlation was used to determine if voluntarily engaging with the optional learning opportunities found in the report card and extra training correlated to answering more quiz questions correctly across the entire sample of learners. – A moderate significant correlation was found between interacting

with the report card and the number of correct answers on quizzes, r(6301) = 0.44, p < 0.001. M

– A strong significant correlation was found between engaging in optional extra training and the number of correct quiz answers, r(6301) = 0.67, p < 0.001.

Research Question Two: Results

• Pearson correlation was used to determine if voluntarily engaging with the optional learning opportunities found in the report card and extra training correlated to answering more quiz questions correctly across the entire sample of learners. – A moderate significant correlation was found between interacting

with the report card and the number of correct answers on quizzes, r(6301) = 0.44, p < 0.001. M

– A strong significant correlation was found between engaging in optional extra training and the number of correct quiz answers, r(6301) = 0.67, p < 0.001.

Pearson's r is a measure of the linear correlation between two variables X and

Y, giving a value between -1 and +1 inclusive, where +1 is total positive correlation, 0 is

no correlation,

Research Question Two: Results

• A comparison of the correct answer streaks per question, which ranged from 0 to 2, for learners in the games condition to those of learners in the no-games condition using a one-way ANOVA. – Learners in the games condition had significantly better correct

answer streaks (M = 1.71) than those in the no-games condition (M = 1.24), p < 0.001.

Research Question Two: Results

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Games OFF Games ON

Aver

age

Corr

ect A

nsw

er S

trea

k Le

ngth

For

Que

stio

ns A

nsw

ered

Tw

ice

Condition

p < 0.001

Limitations

• Two different populations – Two different sizes – Two different company cultures – Unknown efforts to engage employees with platform

• One study – Further investigation is warranted

• More research is required to determine if this result is reproducible and what elements contribute to the results.

Conclusion

• It appears that the opportunity to play a casual game prior to learning positively impacts the level of employee engagement.

• When a learner has a high level of engagement with a gamified platform, they have a higher level of content recall (learning).

1

References

Abdual Jabbar, I, A & Felicia, P. (2015, December) Gameplay engagement and learning in gam-

based learning: A systematic review. Review of Educational Research December, 85(4), 740–779

doi: 10.3102/0034654315577210

Adkins, A. (2016, January 13). Employee engagement in U.S. stagnant in 2015. Resource

document Gallup. http://www.gallup.com/poll/188144/employee-engagement-stagnant-

2015.aspx. Accessed 01 February 2016.

After Taxonomy of Educational Objectives: Handbook I Cognitive Domain (1956). In B.S. Bloom

(Ed.), Committee of College and University Examiners (pp. 201-207). New York: David McKay

Co..

Association for Talent Development [ATD]. (2014). Playing to Win: Gamification and serious

games in organizations.

Cahill, L., & McGaugh, J. L. (1998). Mechanisms of emotional arousal and lasting declarative

memory. Trends in neurosciences, 21(7), 294-299.

Bourne, L. E., Dominowski, R. L., Loftus, E. F. & Healy, A. F. (1979) Cognitive Processes 2nd

Edition. Englewood Cliffs, NJ: Prentice-Hall.

Casual Games Market Report (2007). Casual Games Association

Casual Game White Paper (2008). IGDA. International Game Developers Association.

Current and future prospects for gamification in the education sector, (2015, July 17). Resource

document Emerging Strategy. http://www.emerging-strategy.com/article/current-and-future-

prospects-for-gamified-learning-in-the-education-sector/. Accessed 02 February 2016.

Csikszentmihalyi, M. (1975). Play and intrinsic rewards. Journal of Humanistic Psychology,

15(3), 41-63.

Dicheva, D., Dichev C., Agre G., & Angelova G. (2015). Gamification in Education: A Systematic

Mapping Study. Educational Technology & Society, 18 (3), 75–88.

Dickey, D. M. (2005). Engaging by design: How engagement strategies in popular computer and

video games can inform instructional design. Educational Technology Research and Development,

53, 67–83. doi:10.1007/BF022504866.

2

Eseryel, D., Law, V., Ifenthaler, D., Ge, X., & Miller, R. (2014). An Investigation of the

Interrelationships between Motivation, Engagement, and Complex Problem Solving in Game-

based Learning. Educational Technology & Society, 17 (1), 42–53.

Gagne, R. M. (1972). Domains of learning. Interchange, 3(1), p. 1-8.

Gagne, R. M. (1977). The Conditions of Learning and Theory of Instruction 4th Edition. New

York, NY: Holt, Rinehart and Winston.

Gibson, J. T. (2009). Discussion approach to instruction. In C. M. Reigeluth & A. A. Carr-

Chellman (Eds.), Instructional-design theories and models, Vol. III: Building a common

knowledge base (pp. 99-116). New York, NY: Routledge.

Gao, Y. & Mandryk, R. L, (2011) GrabApple:The design of a casual exergame. In J. Anacleto et

al. (Eds.): ICEC 2011, LNCS 6972 (pp. 35–46). IFIP International Federation for Information

Processing.

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and

practice model. Simulation & Gaming, 33, 441–467. doi:10.1177/1046878102238607.

Gee, J. P. (2007). Learning and games. In K. Salen (Ed.), The ecology of games: Connecting

youth, games, and learning (pp. 21-40). Cambridge, MA: MIT Press.

Gibson, J. T. (2009). Discussion approach to instruction. In C. M. Reigeluth & A. A. Carr-

Chellman (Eds.), Instructional-design theories and models, Vol. III: Building a common

knowledge base (pp. 99-116). New York, NY: Routledge.

Gredler, M. E. (1997) Learning and Instruction: Theory into Practice 3rd Edition. Upper Saddle

River, NJ: Merril, an imprint of Prentice Hall.

Greenfield, P. M. (2010). Video games revisited. In R. van Eck (Ed.), Gaming and cognition:

Theories and practice from the learning sciences (pp. 1-21). Hershey, PA: IGI Global.

Hamann, S. (2001). Cognitive and neural mechanisms of emotional memory.Trends in cognitive

sciences, 5(9), 394-400.

Klemm, W. R. (2007). What good is learning if you don’t remember it? The Journal of Effective

Teaching. 7(1), 61-73.

3

Loreto, I. D., & Gouaich, A. (2010) Social Casual Games Success is not so Casual. RR-10017, pp.

001-011.

Malone, T.W. (1981) Toward a Theory of Intrinsically Motivating Instruction. Cognitive Science.

5(4): p. 333-369.

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomic model of intrinsic

motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning, and instruction:

III. Conative and affective process analysis (pp. 223-253). Hillsdale, NJ: Erlbaum.

Rebetez, C. & Betrancourt, M. (2007, March). Video game research in cognitive and educational

sciences. Cognitie, Creier, Comportament / Cognition, Brain, Behavior. Romanian Association for

Cognitive Science. ISSN: 1224-8398. XI(1), 131-142.

Seels, B. & Glasgow, Z. (1990) Exercises in instructional design. London: Merrill Publishing

Company.

Serious game market size to grow at 16.38% CAGR to 2020 in terms of value say global serious

games research reports focusing on state of play, challenges and industry prospects. (2015,

September 18). Resource document PR Newswire. http://www.prnewswire.com/news-

releases/serious-game-market-size-to-grow-at-1638-cagr-to-2020-in-terms-of-value-say-global-

serious-games-research-reports-focusing-on-state-of-play-challenges-and-industry-prospects-

528208111.html. Accessed 28 March 2016.

Skinner, E., & Belmont, M. (1993). Motivation in the classroom: Reciprocal effects of teacher

behavior and student engagement across the school year. Journal of Educational Psychology,

doi:10.1037/0022-0663.85.4.571

Teachers Surveyed on Using Digital Games in Class. (2013). Resource document Games and

Learning. http://www.gamesandlearning.org/2014/06/09/teachers-on-using-games-in-class/.

Accessed 02 February 2016.

Whitton, N. (2011). Encouraging engagement in game-based learning. International Journal of

Game-Based Learning, doi:10.4018/ijgbl.2011010106

Wouters, P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013, February 4). A

Meta-Analysis of the Cognitive and Motivational Effects of Serious Games. Journal of

Educational Psychology. Advance online publication. doi: 10.1037/a0031311