CS298: HCI Design ClinicsWizard of Oz Prototyping 03/08/10

Dr. Steven Dow

BerkeleyU N I V E R S I T Y O F C A L I F O R N I A

Why Do We Prototype?

Why Do We Prototype?Get feedback on our design fasterExperiment with alternative designsFix problems before code is writtenKeep the design centered on the user

DesignPrototype

Evaluate

Why Do We Prototype?Get feedback on our design fasterExperiment with alternative designsFix problems before code is writtenKeep the design centered on the userCommunicate with otherdesigners, engineers, and clientsBuild confidence in solutionsStay motivated to persevere

DesignPrototype

Evaluate

Prototyping tip #1: Avoid over-investing in prototypes

Sunk-cost reasoning makes design changes difficult

Prototyping tip #1: Avoid over-investing in prototypes

Sunk-cost reasoning makes design changes difficult You buy advance tickets for Wicked the Musical for $50 eachLater you find out the show has received mixed reviews On the day of the show it’s raining and you have a headache

What would you rather do?A) Go see the show, even though you may not enjoy itB) Skip the show, stay home, and watch Wizard of Oz

Topics1. Intro to Wizard of Oz

prototyping2. Exercise: wizarding a wake-

up service3. Dimensions of WOz

prototyping4. Exercise: wizarding a context-

aware application

Wizard of Oz Prototyping

What is the “Wizard of Oz”?

Wizard of Oz (WOz) methods in HCIMake an application operate without (much) code

Must take less time/money than building the real thingGet feedback on the user interface (fidelity matters)

Hi-fidelity interfaces can make users think it’s working Low-fidelity gives users more license to suggest changesInterface is just a façade for a working application

Fake the interaction (like paper prototyping, but the digital interface appears “real”)

An operator uses a separate (sometimes remote) controlinterface to facilitate user interaction

Iterate, iterate, iterate

Examples of WOz methodsNatural language dialog – Travel Agent example

Dahlbäck, Jönsson, Ahrenberg, 1993

Examples of WOz methodsSpeech User Interfaces – SUEDE

Klemmer, Sinha, Landay, 2000

How to make a WOz PrototypePut together an interface “skeleton”Invest minimal time

Create “hooks” in the codeWhere and how the wizard will provide input (e.g., selecting the next screen, entering text, entering a zone, recognizing speech, etc.); Must be possible to replace later with computer

Map out scenarios and application flowPlan out what should happen in response to any user reaction Rehearse wizard functions with a colleague as userMake sure the wizard can perform the task

Prototyping tip #2: Manage users to get desired feedback

Users can respond very differently depending on how the study is conducted

Prototyping tip #2: Manage users to get desired feedback

Users can respond very differently depending on how the study is conductedExperiment on multiple UI designs: Users provide more “damaging” and valuable feedback when given alternatives

Tohidi, Buxton, Baecker, Sellen, 2006

How to test a WOz PrototypeWhat you tell the user mattersTypically they are not told about the human operator (more authentic, but considered unethical in some situations)

Various evaluation methods may work -Think aloud (speak freely as performing tasks)-Retrospective (best when think aloud takes away from app)-Heuristic evaluation (works with experts too)

Give users tasks or a clear indication of what they should be doing Hand-written tasks on paper

Debrief users, reveal wizard

Wizarding a Wake-up Call Service

Wake-up Call ServiceObjectiveCreate and test a service that allows hotel visitors to set/adjust/cancel a wake up call

RulesForm teams of 2 peopleMap out dialog tree and output statementsHandle all possible scenarios Pull 2 people from other teams to “test” the serviceRevise and fix any problems that are discovered20 minutes to write the script, 10 minutes to test with users

Dimensions of Wizarding

Mechanical Turk18th Century Chess Playing Machine

Mechanical TurkLeveraging human intelligence

Wizard design choices (dimensions)Exposure of Wizard What does the user know about the hidden wizard?Task What apart of an application does a wizard emulate?ResponsibilityHow much should the system rely on the wizard?Number of wizardsCan the wizard task be distributed across multiple operators?Expertise of wizardsWhat does the wizard need to know? Can they be amateurs?Location of wizardsAre the wizards co-located or remote?Timing and settingWhen and where are wizard of oz methods appropriate?

Prototyping tip #3: Get the design right and the right design

Dow, MacIntyre, Lee, Oezbek, Bolter and Gandy, 2005

Prototyping tip #3: Get the design right and the right design

Wizard methods can facilitate development throughout a user-centered design process

Dow, MacIntyre, Lee, Oezbek, Bolter and Gandy, 2005

Prototyping tip #3: Get the design right and the right design

Tradeoffs in Wizard of Oz prototyping

Advantages

Disadvantages

Tradeoffs in Wizard of Oz prototyping

AdvantagesFast (faster) and thus, cheaper and more iterative prototypesMore “real” than paper prototypingIdentifies bugs and problems with current designPlaces the user at the center of developmentCan envision challenging-to-build applications Designers learn by playing the wizardDisadvantagesSimulations may misrepresent otherwise imperfect techMay simulate technologies that do not exist (and may never)Wizards may need to be trained and can be inconsistent Playing the wizard can be exhaustingSome system features (and limitations) cannot be simulatedMay not be appropriate in certain venues (e.g., home)

Summary of Wizard of Oz MethodsUse “Wizard of Oz” prototypes to design (and get feedback!) throughout a development process

Learn what users like/dislike, discover problems, and continually evolve the application

Transition towards an application that is fully-functional and user-centered (adjusting the wizard roles and configuration as needed)

Account for potential downsides: inconsistent wizard performance, mismatch of simulation and actual technology, inappropriate for some testing environments…etc.

Wizarding Context-Aware Applications

What is “context-aware”?System senses something about users (or the environment)

Location – where is the user?Identity – who is the user?Proximity – who or what is near the user?Activity – what is the user doing?Situational – what’s happening in the user’s environment? Multi-modal – how is the user interacting with the system?

System then reacts to or prompts the user, or acts on the user’s behalf

How is context sensed?Cameras with image processing Accelerometers for motion sensing, rotationGPS and other indoor location sensorsLight/heat sensors and other environmental sensorsMicrophones for sounds; smell sensors; web sensors; etc.

WOz methods for contextCook’s Collage – Kitchen memory aid

Tran, Calcaterra, Mynatt, 2005

WOz methods for contextTopiary – Location-based applications

Li, Hong, Landay, 2004

Create a context-aware applicationObjectiveCreate and test an application with paper or digital screens that reacts to users using various sensors

RulesForm teams of 3-4 peopleChoose a context and sketch scenariosMap out application flow, create screens to respond to eventsPull people from other teams to “test” the service 30 min to write application, create screenshots 10 min per group to test Show and Tell at 5:40 (you must justify your “sensors”)