(1)

Makahiki and SGSEAM: A Serious Game Framework for Sustainability and Assessment

Method

Yongwen XuCollaborative Software Development Laboratory Department of Information and Computer Science

University of Hawai`i at Mānoa

(2)

Sustainability is “development that meets the needs of the present without compromising the ability of future generations to meet their own needs”.

Brundtland Report, 1987

Introduction

(3)

Sustainability Behavior“We are not always rational decision makers”

Psychology of Sustainable Behavior, Manning, 2009

Energy is “invisible”

Water is “everywhere”

(4)

Sustainability Education “Education as an essential tool for achieving sustainability”

–Agenda 21, Earth Summit 1992, U.N.

“Surprising lack of sustainability curriculum initiatives in institutions of higher education”

– Sendall 2010

(5) Chelsea Hodge, 2010 Campus Sustainability Survey

160+Collegiate Sustainability Competitions

(6)

Information Technology for

Sustainability Education?

(7)

A. Use a "minimal tech" solution such as a web page and manual posting of data and results

Harvard Green Cup

(8)

B. Out-source to a commercial providerOberlin Campus Resource Monitoring

System

Lucid Design Group

(9)

3 weeks competition

C. Build your own custom in-house solution

The Kukui Cup 2011University of Hawai`i at Manoa

2 years software development

(10)

D. Use an Open Source Serious Game Framework

(11)

1. Design a serious game framework for sustainability education

2. Design a method for assessing serious game frameworks

Research Goals

(12)

Serious Game

“a mental contest, played with a computer in accordance with specific rules that uses entertainment to further government or corporate training, education, health, …..”

Zyda, Michael 2005

(13)

Serious Games in Learning and Behavior

MotivationAccomplishment

EnforcementEngagingSocial

(14)

Power House

Reeves, James, 2012

Related work

(15)

Vermontivate

Vermont town energy competition, 2012

Related work

(16)

“a framework comprised of a collection of di erent tools, utilities, and interfaces ffthat hide the low-level details of the various tasks that make up a video game.”

- Sherrod 2007

Game Framework

(17)

Makahiki: An Open Source Serious Game

Framework for Sustainability

(18)

Architecture

Game and Mechanics Library

API

Green Button Data

Wattdepot Data

Manual DataSmartgrid game

Water game

Rafflegame

Badge Quest

Energy game

Authentication

Facebook Integration

GameAnalytics

Resource Manager

MemCache

Database

Responsive Web UI

System Design

Topscoregame

ParticipationReferral

CAS LDAP Django

Score Manager

(19)

What makes Makahiki a Framework?

1. Configurable game elements2. Configurable resources3. Extending new content and games4. Built-in analytics for game evaluation5. Mobile support6. Flexible local or cloud deployment

(20)

1. Configurable Game Elements

(21)

2. Configurable Resource

(22)

3. Extending new content and games

(23)

4. Built-in Analytics for Game Evaluation

(24)

5. Mobile support

Laptop view Mobile view

Responsive Web Design

(25)

6. Flexible local or cloud deployment

(26)

Experience with Makahiki Framework

2012

2013

Upcoming2014

(27)

Instances of the FrameworkUHM’12 HPU’12 EWC’12 HPU’13

Resource Energy Energy Energy & Water

Energy

Smart meters Yes Yes No YesDuration 9 months 3 weeks 2 weeks 3 weeksEligible players 1067 198 129 197Learning actions 144 39 41 39Prizes Yes Yes No YesQuest Mechanics Yes Yes No YesAuthentication CAS LDAP CAS & Internal LDAPDeployment Cloud Local Cloud Local

(28)

Assessing Serious Game Framework

How do we know if a serious game framework is correctly designed ?

(29)

Serious Game Assessment

Related work

(30)

Assessment Tools GEQ (Game Engagement Questionnaire)

QUIS (Questionnaire for User Interaction Satisfaction)

Related work

(31)

SGSEAM: Serious Game Stakeholder

Experience Assessment Method

(32)

Step 1:Plan Assessment• identify

stakeholders• determine

approaches• create schedule

Step 2:Gather Data• carry out

assessment• record data• obtain log data

Step 3:Produce Report• analyze data• interpret

strength and weakness

• produce action document

SGSEAM Overview

(33)

Assessment Vs Evaluation Evaluation:

- is to “determine the quality of a program by formulating a judgment”

» Hurteau et al., Evaluation 2009

SGSEAM is an assessment method- not to judge- not compare with others

(34)

SGSEAM Goals

Identify the strengths and weaknesses of the framework by assessing the stakeholders’ experiences

• for the purpose of self improvement

(35)

SGSEAM Serious Game Stakeholders

Playersparticipate in the game

System adminsinstall and maintain the game infrastructure

Game designersdesign the content and game mechanics

Game managersmanage the game during the game period

Game developersextend, enhance the game

(36)

Serious Game Stakeholders NOT in SGSEAM

Researchers Spectators Funding organizations

(37)

SGSEAM Assessment ApproachesStakeholder Assessment Approach

Player 1. pre-post effectiveness study2. self-reported usability metrics3. engagement metrics

System admin 1. in-lab installation experiment2. post-hoc system admin interview

Game designer 1. in-lab game design experiment2. post-hoc game designer interview

Game manager 1. in-lab game management experiment2. in-game manager interview3. post-hoc game manager interview

Game developer 1. in-lab game development experiment2. post-hoc developer interview

(38)

Multiple Assessment Approaches

Different levels of rigor• in-lab experiment has more data points• post-hoc interview has data in

production

Additive, the more the better

(39)

SGSEAM Evaluation1. Apply SGSEAM to Makahiki

2. Apply SGSEAM to Lucid BuildingOS

(40)

Makahiki Assessment StatusStakeholder Assessment Completed PlannedPlayer pre-post study UH KC 2011

engagement metrics UH KC 2012self-reported usability metrics UH KC 2014

System admin

in-lab installation ICS691 2013post-hoc admin interview HPU KC 2013

Game designer

in-lab game design ICS691 2013post-hoc designer interview HPU KC 2013

Game manager

in-lab game management ICS691 2013post-hoc manager interview HPU KC 2013

Game developer

in-lab game development ICS691 2013post-hoc developer interview UH KC 2014

(41)

Player Pre-post Study Assessment Approach

Energy literacy survey• before and after the game

Energy usage comparison:• before and after the game

(42)

Pre-post Study Result

2011 UH Kukui Cup [2]Energy literacy- 18% (p=0.056) improvement (24 players)

Energy usage:- ~2% reduction (20 teams)

[2] Robert Brewer PhD dissertation

Assessment Outcome:• effectiveness in literacy improvement• not enough evidence in behavior change

(43)

Player Engagement Metrics Assessment Approach

Analyze game log to calculate:•Participation rate•Daily players•Daily play time•Submission rate•Social interaction rate•Game error rate

(44)

Engagement Metrics Assessment Result

2012 UH Kukui Cup (n=1038)

Evidence of successful player engagement

(45)

In-lab experiments

ICS691 Spring 2013• Serious game development course • 8 students (CS major)

• Assessment:- System admin- Game designer- Game developer

(46)

Experiment Design Student assignments:

-Install the framework-Design games-Develop enhancements

Use google form to record:-Time spent-Problem encountered

Write blog posts to describe:-Problems in depth -Experiences

(47)

System admin In-lab installation experiment Result

Avg time (minutes) for installation steps -- 2013 Spring ICS691 (n=8)

Weakness:database configuration, no enough automation

(48)

Game designer In-lab design experiment Result

Avg time (minutes) for design tasks -- 2013 Spring ICS691 (n=8)

Weakness:Smartgrid game predicate design, bugs in design interface

(49)

Game developer In-lab development experiment Result

2013 Spring ICS691 (n=8)

completed “Hello world” widget• 8 students

completed all 5 enhancements• 1 students

Main problem reported• lack of documentation

(50)

Makahiki SGSEAM Assessment Preliminary Outcome

Stake-holder

Assessment Strengths Weaknesses

Player pre-post study

effective in improving energy literacy

no consistent improvement in energy consumption

engagement metrics

game is engaging both in website visit and active interaction

System admin

in-lab installation experiment

relatively easy database configuration, documentation

SGSEAM Assessment Report

(51)

Makahiki SGSEAM Assessment Preliminary Outcome

Stakeholder Assessment Strengths Weaknesses

Game designer

in-lab installation experiment

easy to design most of the games except the smartgrid game

weak UI for specifying smartgrid game predicates; a few bugs.

Game manager

in-lab installation experiment

easy

Game developer

in-lab installation experiment

simple widget is easy to add

lack of documentation

SGSEAM Assessment Report (Cont.)

(52)

Lucid BuildingOS AssessmentTimeline:

Campus Conservation Nationals competition (CCN 2014)

Collaborated with Lucid Design Groupsoftware platform used by CCN

(53)

BuildingOS SGSEAM Assessment Proposal

Stakeholder Assessment approach Expected outcomePlayer pre-post resource usage

comparisoneffectiveness in resource usage reduction

usability online survey identify problem areas in game interface

engagement metrics log analysis

find out the extent of engagement

System admin

post-hoc system admin interview

identify problems during the installation and maintenance

Game designer

post-hoc game designer interview

identify problems in game design interface

Game manager

post-hoc game manager interview

identify problems when managing the game

Game developer

post-hoc developer interview

identify problems in game development

(54)

Lucid Assessment StepsProvide SGSEAM user guide and initial Lucid assessment plan

Identify the stakeholders for BuildingOS

Choose stakeholders from several participants of CCN

Carry out assessment

Produce strength/weakness report

(55)

Anticipated Contributions

1. Makahiki: an open source serious game framework for different organizations to create sustainability related serious games

2. SGSEAM: an method for assessing serious game framework

3. Insights into the real world experiences of using serious game framework for sustainability education

(56)

Anticipated Contributions (cont.)

4. Evidence regarding the strengths and weaknesses of Makahiki

5. Evidence regarding the strengths and weaknesses of BuildingOS

6. Evidence of strengths and weaknesses of SGSEAM

(57)

Publications from this research1. SGSEAM: Assessing serious game frameworks from a stakeholder experience perspective (Gamification 2013)

2. Energy Feedback for Smart Grid Consumers: Lessons Learned from the Kukui Cup (Energy 2013)

3. Makahiki+WattDepot: An open source software stack for next generation energy research and education  (ICT4S 2013)

4. Beyond kWh: Myths and fixes for energy competition game design  (Meaningful Play 2012)

5. Lights Off. Game On. The Kukui Cup: A Dorm Energy Competition  (CHI 2011 Workshop on Gamification)

(58)

Time lines (completed)Time TasksJanuary 2010 Joined Makahiki v1 (non-framework)

developmentOctober 2011 UH Kukui Cup’11

November 2011 Makahiki v2 (framework) development

September 2012 HPU, EWC, UH Kukui Cup’12

March 2013 In-lab SGSEAM assessment using ICS691

September 2013 HPU Kukui Cup’13

(59)

Time lines (planned)Time TasksDecember 2013 Post-hoc interview assessment using

HPU’13 instanceFebruary 2014 Player self-reported (in-game survey)

assessment using UH’14 instanceMarch 2014 Developer post-hoc interview

assessment using UH’14 instanceApril 2014 SGSEAM assessment of Lucid

BuildingOS using CCN 2014 instancesAugust 2014 Dissertation defense

(60)

Mahalo !yxu@hawaii.edu

Makahiki Project: http://makahiki.io