experimental game prototyping and play testing using iterative design

experimental prototyping and play testing using iterative game

design

experimental prototyping and play testing using iterative game

design

2010

Lecture in the course International Game Production Studies I

Mirjam Palosaari EladhariGotland University, Sweden

2010

Lecture in the course International Game Production Studies I

Mirjam Palosaari EladhariGotland University, Sweden

Overview

- Wicked problems and the importance of building

- Types of Prototypes

- Types of Play Tests

- Obtaining Data

- Trade-offs

Mirjam P. Eladhari, [email protected] Gotland University, Department of Game Design, Technology and learning

Focus of this talk:

experimental prototyping and play testing using iterative game design

how to plan for obtaining data from play tests that can yield answers to specific research questions.

Read more: Design for Results: Considerations for experimental prototyping and play testing using iterative game design by Mirjam P Eladhari and Elina M I Ollila. Article to be published in a special issue of the Journal Simulation and Gaming focusing on game research methods. (Uploaded to course website)


Prototype

A prototype is “played, evaluated, adjusted and played again, allowing the designer or design team to base decisions on the successive iterations or versions of the game.

Iterative design is a cyclic process that alternates between prototyping, play-testing, evaluation, and refinement.”

Salen and Zimmerman (2001)


Early questions

• What types of game play dynamics and game play experiences can a certain, mechanic, feature, approach or method result in?1

• What are the qualities of the play experience, and how can these qualities be documented, analysed, interpreted and explained comprehensively?

• How can it be decided if a certain game play feature results in something valuable, such as a new type of experience, a meaningful experience, or a ‘better experience’ in some other way, and if so, compared to what? Researchers may find themselves in situations where innovative results do not necessarily lend themselves to comparisons.

• What type of data can be useful to obtain in order to explore the research question?


Beware• Researchers need to, in the design

process, take into account what questions they aim to explore and stay focussed on these throughout the design and implementation work.

• It is easy to fall into a frame of mind where one aims to produce a good game, losing focus of obtaining research material.

• In the process (especially when implementing) take a step back regularly to think about the research question.


Wicked Problems

‘wicked problem’ is used in social planning to describe problems where every attempt at producing a solution changes the understanding of the problems

(Rittel & Webber, 1973)


Wicked problems in game design

“For a wicked problem such as game design, exploring design space consists of navigating the complex relationships and constraints among individual design features, while at the same [time] discovering or inventing new features and approaches that expand the design space.

All existing games form tiny islands of partially understood regions of design space; all around these islands lays a vast ocean of unexplored potential design space waiting to be brought into existence through the invention of new features and approaches, and mapped out through the hard empirical work of exploring a variety of designs.”

Mateas and Stern (2005)


Build it to understand it

“[...] if game studies is limited to analysing existing games and design spaces, it can be problematic to imagine or theorise about potential game features outside of these design spaces. Models about the nature of games and their features run the risk of being incomplete or wrong, simply because certain design spaces have not yet been explored.”

(Mateas and Stern 2005)


Navigational Aid in the wicked problem space of

game design

- Types of Prototypes

- Types of Play Tests

- Obtaining Data


TYPES OF PROTOTYPESTYPES OF PROTOTYPES

Prototyping by acting and showing

• body storming - participants imagine the game and act as though it would exist, in either a real or imaginary place where it could be played. (Burns, Dishman, Verplank, & Lassiter, 1994)

• polished video scenarios, the purpose of which is to show how a concept would work in its natural settings.

FIGURE The player is taking a picture of a person smiling, 2. The smile is mapped to a smiley in a grid that the player needs to complete (i.e., take

pictures with people having similar expressions on their faces as the smileys do)

Early


Paper (or physical) prototypes

• A physical prototype can at an early stage give pointers to whether a designed game mechanic results in the intended game dynamics when played. A physical prototype is often made of paper mock-ups of the intended game, but can also include physical ‘bits’ such as figurines and tokens.

game-mastering method, where the player was brought to various game situations with the help of a game master, like in pen-and-paper role-playing games.

Middle


Computationally aided physical prototypes

• In cases where the game mechanics demand higher degrees of computation of significant values it can be useful to add aids for calculation to a paper prototype.

• A common and very simple method is to use excel sheets

Middle


The developers of Raptor found that using a table-top was superior for collaborative sketching compared to the interface of traditional PCs. Designers were presented with a sand-box environment where they could shape the geography using hand-gestures. They could also ‘stamp’ game bits into the environment such as cars for a racing game, and attach a ‘camera’ to the object representing the player, giving the user, who had access to a PC client interface, their point of view in the geography.

Raptor, a tool for sketching and

prototyping games using a table-top surface.

Raptor: Sketching Games with a Tabletop Computer, by JD Smith, FuturePlay 2010Mirjam P. Eladhari, [email protected] Gotland University, Department of

Game Design, Technology and learning

Software Prototypes

• Software prototypes are quite often thought to be something that is created later in the project, when there is already game design documentation available. However, plenty of fast prototyping tools are available, ranging from general purpose tools like Flash to more specialised frameworks.

Late


Production cycle, example


TYPES OF PLAY TESTSTYPES OF PLAY TESTS

ad-hoc test

An ad-hoc test is a quick informal test which requires minimal organisation. It can be as simple as finding a colleague not working on the same project to quickly test some aspect of the prototype on them.

(grab someone in the corridoor- test)


focus testIn a focus test a group of potential players are probed about their perceptions, opinions, beliefs and attitudes towards the prototype. Questions are asked in an interactive group setting.

(put a group of people from your target group in the same room and ask them your questions)


expert evaluation

An expert evaluation is the appraisal of a prototype by someone who has the professional training or experience to make an informed judgement on the design.

(Show what you have to an expert, get their advice)


QA TestThe functional test, or quality

assurance (QA), can be conducted by the production team or by QA-experts. QA is conducted in order capture errors in the functionality of the prototype as well as for balancing game play according to the intended game play experience

(Find the bugs and fix them)Mirjam P. Eladhari, [email protected] Gotland University, Department of

Game Design, Technology and learning

Participatory designParticipatory design workshops are conducted by

the production team and a small number of invited guests, potentially experts in the field or potential players of the game.

The workshops are often intended to aid in balancing the game and to eliminating dysfunctional elements or features. They can also be useful for gathering new ideas for further iterations or for enhancing the quality the prototype according to the goals by specific investigations.

(Have people your trust go through the test with you, and get their advice on your design as you go along)


Guided scenariosIn guided scenarios a test-leader may

use the Wizard of Oz - method to simulate user-interaction. The scenarios designed for the particular prototype are played individually by participants(Wizard of Oz experiment is a research experiment in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being.)

(Game master what you haven’t implemented. If it doesn’t work you save lots of time knowing that.)


Example (4 roles)

The Player, the Game master, the Engine, and the Manual (taking the pictures)


game-mastered play sessions

In game-mastered play sessions several participants may interact with a prototype, or scenarios designed for the particular prototype, guided by a game master.

Similar to guided scenarios, but later, when the prototype is functional. Ie, no wizard-of-ozzing for functionality.


free-form play sessions

In free-form play sessions participants are interacting with the prototype unaided by guiding test-leaders or game-masters.(common end-phase play test. Put the user in front of the screen and see what happens. ...Or send it off to a testing consultant along with your testing script.)


Practical Considerations

Type of play test Who testsTiming in production-cycle

Ad-hoc test

TeamColleaguesConsultants

very early

Participatory design workshops early

Quality Assurance (function-test) medium

Expert evaluations early-medium

Focus test

Players fromtarget group(s)

very early

Guided scenarios early

Game mastered play sessions medium

Free-form play sessions late


OBTAINING DATAOBTAINING DATA

Surveys?Surveys can be used to capture

players’ attitudes, get ideas for development and can also be used for a ‘memory check’ - in those cases a player remembers or have forgotten something about the design

The effort required for this method is less than for many other methods for obtaining data.


Example – Surveys as stand-alone tool


Interviews?

Interviews can allow for obtaining data that might not arise in surveys.

In some cases it may be useful to add a survey in addition to the interview in order to verify the results and to see if some results were received for ‘pleasing’ the interviewer, that is, give answers based on their interpretation of what they think the test leader wants to hear


Teach back tasks

‘teach-back tasks’ – players are asked to describe the system they tested to someone who has no prior knowledge of it.

The player ‘does the job’

Can be used both in surveys and interviews.Teach-back tasks are often used in order to assess users’ mental models of

systems (Veer, Wijk, & Felt, 1990; Puerta-Melguizo, Chisalita,& Veer, 2002).

For example, in a test of a prototype where the players’ avatars had different ‘moods’ given by a psychological model, players were asked to explain to a friend how the mood affected what they could do or not do in the game world prototype (Eladhari, 2009).


Video tape it?

Is it is useful to videotape the interaction?

Possible to later on make verbatim transcriptions of what players said in the test.

Allows for detailed study of body-language, actions and utterances.


Bio feedback?Is it useful to obtain physical data

from the participants of the test, and if so, what data is most likely to be useful?

Potential physical data include:

• eye-tracking,

• heart-rate variability,

• galvanism and temperature in fingers,

• electroencephalography (EEG) or f MRI,

• posture (posture sensors may be placed on participants’ chairs, see (Plass, Perlin, Nordlinger, & Isbister, 2010)


Data mining?

If the test involves a software prototype, it can be useful to also develop or use a back end-system for capturing events in the form of player-input to the system.

Consider:

• to what extent it is useful to log the player's interactions with the system,

• which types of interactions to log.


Data mining exampleThe team at Microsoft Labs who conducted the testing of Halo 3 maps showing recurring events mapped to the geography of the game. This was useful for the designers of the game when they iteratively redesigned the environment and the placement of items in it relevant to the game play.

Thompson, C. (2007). Halo 3: How Microsoft Labs invented a new science of play. Wired Magazine (15.09).

Map used at Microsoft Game Labs showing a top down view of the fourth mission of Halo 3. The dots show the locations of player deaths. (thanks to John Hopson and MS game labs for permission to use

picture)


Process the data

• Is it useful to make a rough coding scheme, for example counting certain types of utterances, behaviours, and/ or expressions or reactions?

• Is it useful to make verbatim transcriptions of the videotaped play tests? If so, is it useful to not only transcribe the utterances but also (or in- stead) make notes of facial expression, body language, laugher etc. to the transcripts?

• Is it useful to trace and describe the actions a player performs in relation to the paper prototype? For example it can be useful to observe and describe in what order players do something, and if there are patterns in how they approach different elements of the prototype.


Obtaining Data

Data to obtain

Asking players via: Observing players’:

Surveys Interviews

Interaction - Choice of actions- Patterns of actions

Expressions - Facial expressions- Body language- Voice (pitch and tone)- Verbal expressions (think-aloud, conversation, etc)

Physiological responses - Heart rate variability- Gaze (eye tracking)- EEG- Temperature and galvanism in fingers

Capturing of data

(documentation inherent in the actual survey forms)

- Note taking- tape recording- video taping

- Video tapingIn case of a software prototype:- Screen capture of video sequences- Logging of significant actions

- Video-taping- sound recording

- Observation ofmeasured values in real-time and note-taking- Log-reports from devices measuring physiological responses.

Processingof obtained data

- Identification of relevant parts of documentation.- Verbatim transcriptions of relevant parts, and annotation of significant expressions.- Synchronisation of used reports of documentation from different sources, such as software event-logs, verbatim transcripts and log reports of physiological responses. After this the data may be analyzed and interpreted with methods appropriate for the individual project


The trade-off of research freedom vs resources

Game Industry Game Hobbyists

Game Research

Success criteria

Make good games. Sell them.

Pursue own passionate interest.

New knowledge. Understand games and

understand how to make better games.

Resources Large Small Medium

Testing Yes – rigorous Seldom Yes – rigorous

Freedom to share knowledge No – constrained YES YES


Navigation

• 1. Question.

• 2. What kind of prototype can answer it?

• 3. What data is needed to proove/refute?

• What tests to make?

• How process the data?


Thank you for listening

Thank you for listening

experimental game prototyping and play testing using iterative design

Education

department of game design

iterative design

design process

game studies

design practice

design team

game designfor

good game