sculpting the augmented world: scale and narrative in augmented ux design

7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design

1/36

Thank&you&for&having&me&here&today.&This&morning&Im&going&to&present&some&ideas&Im&

working&through&about&designing&user&experiences&for&a&world&where&many,&if&not&all&

objects&in&your&environment&have&some&form&of&networked&digital&component.&


2/36

Let&me&begin&by&telling&you&a&bit&about&my&background.&Im&a&user&experience&designer.&I&

started&designing&Web&sites&back&in&the&Jurassic.&Heres&a&design&I&worked&on&for&HotBot,&an&

early&search&engine.&


3/36

Ive&also&worked&on&the&user&experience&design&of&a&lot&of&consumer&electronics&products&

from&companies&youve&probably&heard&of.&


4/36

Along&the&way&I&wrote&a&couple&of&books&based&on&my&experience&as&a&designer.&One&is&a&

cookbook&of&user&research&methodsthe&second&edion&came&out&last&falland&the&

second&describes&what&I&think&are&some&of&the&core&concerns&when&designing&networked&

computaonal&devices.&It&came&out&a&couple&of&years&ago.&The&ideas&for&this&talk&originated&

in&the&second&book.&


5/36

I&also&started&a&couple&of&companies.&The&first,&Adapve&Path,&is&primarily&focused&on&design&

for&the&Web.&With&the&second&one,&ThingM,&I&got&deep&into&developing&hardware.&


6/36

ThingM&primarily&designs&and&manufactures&a&range&of&RGB&LEDs&with&aached&controllers&

for&architects,&industrial&designers&and&hackers.&These&have&been&on&sale&for&about&five&

years&and&theyre&used&in&everything&from&flying&robot&prototypes&to&costumes&in&Lady&

Gagas&stage&show.&


7/36

We&also&spent&some&me&exploring&the&potenal&uses&of&augmenng&everyday&objects&with&

embedded&technology.&

&

On&the&leU&is&an&RFID&wine&rack&thats&faceted&classificaon&browser&to&a&cloudbased&

service.&For&example,&it&can&show&you&all&the&2003&Napa&Cabernet&Sauvignons&which&are&ready&to&drink&and&whose&current&market&price&is&under&$50.&On&the&right&is&a&capacive&

sensing&kitchen&cabinet&knob&that&glows&when&you&touch&it.&Its&&an&exploraon&in&making&a&

digital&product&that&would&sll&be&useful&20&years&aUer&it&was&made.&

&


8/36

In&January&of&this&year&I&joined&PARC&as&a&principal&in&its&Innovaon&Services&group.&


9/36

But&I&didnt&come&here&to&talk&about&myself.&&When&I&was&last&here&two&

years&ago,&I&advocated&renaming&augmented&reality&as&the&less&excing

sounding,&but&perhaps&more&accurate&somac&data&percepon.&


10/36

My&argument&was&that&visual&AR,&informaon&layered&onto&our&visual&field,&faces&a&big&

problem,&and&thats&the&limited&bandwidth&of&human&percepon.&We&can&start&with&how&

lowresoluon&our&eyes&are.&This&is&our&foveal&resoluon,&which&you&will&noce&wouldnt&

even&cut&it&as&the&cheapest&mobile&phone.&


11/36

To&compensate,&our&eyes&move&in&saccades&which&last&between&20ms&and&200ms,&or&5&to&50

mes&per&second.&


12/36

So&this&leads&to&a&back&of&the&envelope&calculaon&of&eye&bandwidth&between&100&bits&per&

second&and&10K&bits&per&second&

&

Thats&around&4&orders&of&magnitude&slower&than&a&modern&frontside&bus.&&

&The&brain&deals&with&this&through&a&series&of&ingenious&filters&and&adaptaons&to&create&the&

illusion&of&an&experience&of&all&reality,&but&at&the&core&theres&only&a&limited&amount&of&

bandwidth&available&and&our&visual&senses&are&easily&overwhelmed.&And&unfortunately&our&

eyes&and&brains&are&not&subject&to&Moores&Law.&Were&not&going&to&get&more&foveal&

resoluon&or&cognive&processing&capability&any&me&soon.&


13/36

My&argument&is&to&broaden&the&definion&of&AR&to&be&the&experience&of&any&kind&of&data&by&

any&sense,&to&change&our&language&away&from&augmented&reality&as&primarily&visual&

augmentaon&to&something&thats&more&representave&of&the&whole&experience&of&data&in&

the&environment,&the&bodily&experience&of&the&informaon&shadows&that&objects&we&

encounter&in&the&world&cast&into&the&internet,&the&personal&experience&of&the&Internet&of&

Things.&I&called&this&Somac&Data&Percepon.&


14/36

Once&we&broaden&our&definion&to&include&other&senses,&we&can&start&creang&hierarchies&

of&informaon&that&match&the&capabilies&of&our&senses&and&avoid&pung&secondary&

informaon&into&our&primary&sense.&We&can&begin&associang&secondary&data&with&

secondary&senses.&


15/36

There&are&plenty&of&ways&we&can&augment&the&world&that&doesnt&require&increasing&the&

load&on&our&visual&cortex.&Were&already&sufficiently&experienced&with&phone&buzzing&that&

we&can&feel&phantom&vibraons&because&our&brains&have&deeply&embedded&that&sensaon&

as&meaningful&and&when&theres&a&misfire,&we&jump.&Other&projects,&such&as&Jason&Wilsons&

Audio&Perfume&create&shiUing&soundscapes&that&are&ed&to&informaon.&You&can&think&of&

other&examples,&such&as&an&armband&that&gets&warmer&as&you&bike&past&an&apartment&thatsfor&rent&in&your&price&range.&These&arent&nearly&as&high&resoluon&signals&as&visual&ones,&

which&makes&them&feel&less&important,&but&I&think&theyre&actually&incredibly&important&

because&of&the&informaon&they&remove&from&the&visual&channel.&


16/36

Next,&I&want&to&say&that&Im&a&firm&believer&that&everything&is&going&to&be&augmented&with&

networked&informaon&processing&very&soon.&


17/36

For&example,&adidas&has&the&miCoach&and&miCoach&elite&systems,&in&which&a&soccer&ball,&the&

players&shirts,&their&shoes&and&somemes&the&space&theyre&in&are&all&networked&and&

instrumented.&The&screen&devices&they&carry&are&the&displays&of&the&informaon&shadows&

all&of&the&things&are&casng&onto&the&Net.&Very&soon&this&is&going&to&be&your&office,&your&

playground,&your&home&and&your&coffee&shop.&Everything&that&can&be&connected&to&the&net,&

including&things&that&probably&shouldnt&be,&will&be.&Full&stop.&


18/36

Now&we&get&to&the&interesng&user&experience&challenge.&How&do&we&design&for&the&

environment&where&everything&has&a&data&shadow&and&can&be&perceived&through&any&

sense?&Designers&work&well&with&constraints,&so&Im&focusing&on&two&ideas.&The&first&is&that&

the&scale&of&an&experience&is&a&crical&component&of&how&that&experience&is&perceived.&


19/36

Computers&have&tradionally&had&interfaces&that&are&roughly&sized&to&our&bodies.&If&you&

look&at&a&laptop,&the&inputs&are&the&size&of&your&hands&and&the&output&is&on&the&scale&of&your&

head.&We&dont&have&great&models&for&designing&experiences&for&different&scales,&and&when&

we&try&to&map&what&we&know&from&designing&PC&experiences&directly&onto&other&scales,&it&

fails,&as&everyone&who&has&ever&designed&a&phone&interface&for&the&first&me&knows.&

&

At&PARCs&ubicomp&program&in&the&late&80s&and&early&90s&they&defined&the&tab,&pad&and&

board&as&names&for&the&scales&of&the&devices&they&were&developing,&leveraging&the&form&

factors&of&office&supplies&that&had&proven&to&be&at&useful&sizes.&


20/36

This&is&the&scale&Ive&been&using.&Its&of&course&inspired&by&the&Eames&Powers&of&Ten&films,&

but&Ive&been&using&it&to&understand&the&constraints&and&potenal&of&devices&at&different&

scales.&Screens&dont&work&interacvely&when&you&approach&the&covert&scale,&which&is&why&

wristmounted&devices&can&at&best&be&output&devices&displaying&a&limited&set&of&visual&

informaon,&but&theyre&great&for&sending&a&small&message&no&one&else&knows&about.&Thats&

really&hard&to&do&with&a&big&display&such&as&a&TV.&Buons&dont&work&well&controlling&experiences&on&the&environmental&scale&and&above,&because&theyre&too&small&relave&to&

the&object.&


21/36

The&two&screen&experience,&that&mythological&consumer&electronics&grail,&is&really&a&

problem&of&mismatching&scales.&How&can&you&create&a&single&experience&that&

simultaneously&spans&the&mobile&and&personal&scale?&


22/36

To&me&this&brings&me&to&my&second&idea,&which&is&that&for&an&experience&to&make&sense&

across&various&scales,&we&need&a&narrave&to&e&it&together.&I&mean&this&quite&literally.&We&

need&a&story&of&some&sort&that&will&help&us&make&sense&of&why&what&were&doing&at&any&

given&moment&maers.&The&desktop&metaphor&is&a&kind&of&a&narrave.&When&it&was&applied&

to&the&same&kinds&of&funcons&that&had&existed&with&the&command&line,&it&made&them&much

easier&for&people&to&understand&because&it&told&a&story&about&office&work.&It&wasnt&a&perfect&mapping,&because&it&was&a&metaphor,&but&it&was&beer&than&what&it&replaced.&


23/36

Look&at&the&way&we&explain&what&future&technology&is&going&to&do.&These&days&its&always&in&

the&form&of&stories&that&use&the&technology,&whether&its&being&a&caring&parent&while&

engaged&in&some&kind&of&serious&mulnaonal&project,&or&opmizing&your&crop&planng,&its&

always&a&story&about&what&the&end&result&is&going&to&be.&


24/36

Reeves&and&Nass,&two&Stanford&professors,&did&a&series&of&groundbreaking&experiments&that&

showed&that&no&maer&how&you&cut&it,&when&people&are&in&dialog&with&a&device,&they&treat&

it&like&a&person.&And&since&the&kinds&of&relaonships&we&have&with&people&are&essenally&

narraves,&this&in&effect&means&that&all&of&our&experiences&with&devices&have&a&narrave&

component,&whether&we&put&one&in&or&not.&


25/36

And&so&we&come&to&transmedia&storytelling.&


26/36

Many&of&you&are&probably&familiar&with&Henry&Jenkins&work&on&how&stories&are&now&told,&

with&bits&and&pieces&scaered&across&many&media.&Jenkins&canonical&examples&is&the&

Matrix,&where&characters&and&acon&in&the&second&and&third&film&didnt&really&make&sense&

unless&you&had&watched&the&Animatrix&cartoon,&read&the&comics&and&played&the&games.&&A&

more&recent&example&is&Jorgits&and&the&end&of&Winter,&which&is&a&set&of&childrens&stories&

that&are&told&in&an&app&thats&mostly&a&novel,&but&has&elements&of&video&games;&however,&in&addion&to&that&the&characters&have&their&own&Twier&feeds,&Facebook&pages,&Soundcloud&

streams&and&Pinterest&pages.&The&experience&is&carried&across&devices&and&media.&


27/36

In&the&same&way&that&you&can&keep&watching&a&nelix&movie&as&you&jump&from&device&to&

device.&


28/36

Or&read&a&Kindle&book&everywhere.&


29/36

So&heres&my&thought.&That&the&way&that&we&can&bridge&the&gap&in&our&ability&to&create&

compelling&user&experiences&that&span&scales&is&through&using&narrave&to&tell&stories&that&

link&together&the&individual&acons&across&devices&and&between&screens.&I&think&that&this&is&

already&happening.&


30/36

One&of&my&favorite&ubicomp&products&from&the&toy&world&is&five&years&old&and&was&a&

commercial&failure,&but&I&think&it&points&to&whats&possible.&It&was&called&Clickables&and&it&

was&developed&for&the&Disney&Fairies&lines&by&a&Hong&Kong&company&called&TechnoSource.&It

linked&the&virtual&world&Pixie&Hollow,&the&Fairies&virtual&social&network&and&chat&space,&with

the&physical&world&of&the&girls&who&were&the&target&audience&through&the&use&of&smart&

jewelry.&

&

Source:&Disney&Clickables&


31/36

When&two&kids&put&their&Clickables&bracelets&together,&their&avatars&would&link&up&in&Pixie&

Hollow.&Moreover,&all&of&the&products&had&simultaneous&onlineoffline&existence,&so&for&

example,&when&a&girl&got&one&of&the&charm&bracelets,&she&could&touch&the&SBconnected&

jewelry&box&and&her&fairy&avatar&gets&a&version&of&the&same&charm,&plus&some&unique&magic&

buerflies&she&could&only&get&by&geng&one&of&these&charms,&or&being&friends&with&

someone&who&did.&


32/36

Now&the&execuon&of&the&Clickables&experience&leU&much&to&be&desired,&but&I&think&that&the&

basic&components&of&weaving&the&augmented&physical&device&with&the&social&network&was&

powerful.&The&mode&of&use&explained&how&the&charms&and&the&jewelry&worked&provided&a&

jusficaon&for&why&to&use&it.&The&narrave&is&a&kind&of&scaffolding&on&which&the&experience&

is&built.&


33/36

In&the&grownup&world,&we&have&this&collaboraon&between&Diageo&and&Evrythng,&a&Swiss&

Internet&of&Things&company.&Boles&of&a&giU&whiskey&have&a&unique&QR&code&that&leads&to&

an&app&that&allows&the&buyer&to&record&a&video&for&the&recipient.&The&technologywhy&its&

there&and&how&to&use&itis&embedded&in&a&narrave.&Its&a&giU.&Youre&not&going&to&drink&

this&special&bole,&its&for&giving,&and&heres&some&technology&to&help&with&that.&


34/36

This&is&Good&Night&Lamp,&a&set&of&paired&augmented&objects.&The&name&itself&creates&an&

implicit&narrave&that&tells&you&how&to&use&these.&The&narrave&is&what&drives&the&design,&

the&interacon&and&what&jusfies&the&technology.&


35/36

So,&in&conclusion,&we&need&to&think&about&transmedia&narrave&as&a&means&to&unify&

augmented&world&experiences&across&devices,&across&uses,&and&across&scales,&to&bridge&the&

device&disconnects&that&vex&us&and&which&we&call&second&screen&experiences&and&

appcessory&experiences.&As&designers&its&our&jobs&to&tell&people&why&and&how&they&need&to&

use&our&technology.&In&effect,&we&already&tell&stories,&but&I&think&were&doing&a&terrible&job&

of&telling&stories&about&the&augmented&world,&and&why&its&not&just&important,&but&crical&that&people&understand&and&engage&with&it.&We&didnt&need&the&desktop&metaphor&forever,&

but&it&certainly&helped&during&a&crical&period&of&compung&to&give&people&a&structure&with&

which&to&understand&the&potenal&of&computers&and&how&they&could&take&advantage&of&

that&potenal.&Similarly&we&need&new&stories&to&tell&about&our&devices&today,&we&need&

stories&about&the&augmented&world.&

&

Star&Wars&meline&(CC)&by&williampcoleman:&hp://www.flickr.com/photos/20919058@N05/5850117502/&


36/36

Thank&you&

sculpting the augmented world: scale and narrative in augmented ux design

Documents