the ux of predictive behavior for the iot (2016: o'reilly designing for the iot)
TRANSCRIPT
Good afternoonandthanksforhavingmehere.InthistalkIwanttolookatthedesign
challengesofsystemsthatanticipateusers’needsandthenactonthem.Thatmeansitsitsat
theintersectionoftheinternetofthings,userexperiencedesignandmachinelearning,and
althoughpeoplehavedealtwithoneofthosedisciplinesbefore,Idon’tthinkthey’veever
beencombinedinquitethewaystheyarenow,orwiththecurrententhusiasm.
Thetalkisdividedintoseveralparts:itstartswithanoverviewofhowIthinkInternetof
Thingsdevicesareprimarilycomponentsofservices,ratherthanbeingself-contained
experiences,howpredictivebehaviorenableskeycomponentsofthoseservices,andthenI
finishbytryingtotoidentifyuserexperienceissuesaroundpredictivebehaviorand
suggestionsforpatternstoamelioratethoseissues.
Acoupleofcaveats:
- Mycurrentworkinthisfieldfocusesalmostexclusivelyontheconsumerinternetofthings,
soIseemostthingsthroughthatlens.PredictiveAIhasalonghistoryinindustrial
applications,it’sintheconsumerspacethatwereallythetheUXissues.
- IwanttopointoutthatfewifanyoftheissuesIraisearenew.Thoughtheterms“internet
ofthings”and“machinelearning”arehotrightnow,theideashavebeendiscussedin
researchcirclesfordecades.Searchfor“ubiquitouscomputing,”“ambientintelligence,”and
“pervasivecomputing”andyou’llseealotofgreatthoughtinthespace.Ifyou’rereally
ambitious,youcanreadtheArtificialIntelligenceandCyberneticsworksofthe50sand60s
andyou’llbesurprisedbytheprescienceofthepeopleworkinginthisspacewhentheentire
world’scomputepowerwasaboutasmuchasmykeyfob.
- Therearealotofideashere,andIwillalmostcertainlyunder-explainsomething.ForthatI
apologizeinadvance.Mygoalhereistogiveyouageneralsenseofhowthesethepieces
connect,ratherthananin-depthexplanationofanyoneofthepieces.
- Finally,mostofmyslidesdon’thavewordsonthem,soI’llmakethecompletedeckwitha
transcriptavailableassoonI’mdone.
0
Let me begin by telling you a bit about my background. I�m a user experience designer. I was one of the first professional Web designers. This is the navigation for a hot sauce shopping site I designed in the spring of 1994.
1
I’vealsoworkedontheuserexperiencedesignofalotofconsumerelectronics
productsfromcompaniesyou’veprobablyheardof.
2
Iwroteacoupleofbooksbasedonmyexperienceasadesigner.Oneisacookbookof
userresearchmethods,andtheseconddescribeswhatIthinkaresomeofthecore
concernswhendesigningnetworkedcomputationaldevices.I’malsomarriedtoone
oftheauthorsofthisbook,sothinkingabouttheimpactofthedesignofconnected
devicesonpeopleiskindofafamilybusiness.
3
Ialsostartedacoupleofcompanies.Thefirst,AdaptivePath,wasprimarilyfocused
ontheweb, andwiththesecondone,ThingM,Igotdeepintodevelopinghardware.
4
TodayIworkforPARC,thefamousresearchlabthatinventedthepersonalcomputer,
objectorientedsoftware,thetabletcomputer,andlaserprinter,asaprincipalinits
InnovationServicesgroup.Wehelpcompaniesreducetheriskofadoptingnovel
technologiesusingamixofsocialresearch,designandbusinessstrategy.
5
IwantstartbyfocusingonwhatIfeelisa keyaspectofconsumerIoTthat’soften
missedwhenpeoplefocusonthehardwareoftheIoT,whichisthatconsumerIoT
productshaveaverydifferentbusinessmodelthantraditionalconsumerelectronics.
6
Historically,acompanymadeanelectronicproduct,sayaturntable,theyfound
peopletosellitforthem,theyadvertiseditandpeopleboughtit.Thatwas
traditionallytheendofthecompany’srelationshipwiththecustomeruntilthat
personboughtanotherthing,andallofthevalueoftherelationshipwasinthe
device.WiththeIoT,thesaleofthedeviceisjustthebeginningoftherelationship
andphysicalthingholdsalmostnovalueforeitherthecustomerorthemanufacturer.
7
When you have a multitude of connected devices and apps, value shifts to services and the devices, software applications and websites used to access it—its avatars—become secondary. A camera becomes a really good appliance for taking photos for Instagram, while a TV becomes a nice Instagram display that you don’t have to log into every time, and a phone becomes a convenient way to check your friends’ pictures on the road.
Hardware, physical things, become simultaneously more specialized and devalued as users see “through” each device to the service it represents. The avatars exist to get better value out of the service.
8
Amazon reallygetsthis.Here�satellingolderadfromAmazonfortheKindle. It’s
saying�Look,usewhateverdevice youwant.Wedon�tcare,aslongyoustayloyaltoourservice.Youcanbuyourspecializeddevices,butyoudon�thaveto.�
9
WhenFirewasreleased5yearsago,JeffBezosevencalled itaservice.
10
Mostlarge-scaleIoT productsareserviceavatars.Theyusespecialized sensorsand
actuatorstosupportaservice,buthavelittlevalue—ordon’tworkatall—without
thesupportingservice.SmartThings,whichwasacquiredbySamsung, clearlystates
itsserviceofferingrightupfrontontheirsite.Thefirstthingtheysayabouttheir
productlineisnotwhatthefunctionalityis,butwhateffecttheirservicewillachieve
fortheircustomers.Theirhardwareproducts’functionality,howtheywilltechnically
satisfytheservicepromise,isalmostanafterthought.
11
Compare thattoX10,theirspiritualpredecessorthat’sbeeninthebusinessfor30
years.AllthatX10tellsisyouiswhatthedevicesare,notwhattheservicewill
accomplishforyou.Idon’tevenknowifthereISaservice.WhyshouldIcarethat
theyhave“modules”?Ishouldn’t,andIdon’t.
12
Simplyconnectingexistingstufftotheinternetdoesnotproduce customervalue…
13
Simpleconnectivityhelpswhenyou’retryingtomaximizetheefficiencyofafixed
process,butthat’snotaproblemthatmostpeoplehave.We’vebeenabletosimply
connectvariousdevicestoacomputersinceaTandyColorComputerscouldlightsoff
andonoverX10in1983.TodayyoucanbuyamodulefromParticle,ElectricImpora
dozenothercompaniesandintegrateitinamonthtoconnectanyarbitrarydeviceto
theInternet.Theproblemisthatthatwasn’tveryusefulthen,andit’snotveryuseful
now.IfyoureplacetheTandywithaniPhoneandthelampwithawashingmachine…
14
…oraneggcarton,youstillhavethesameproblem,andit’sauserexperience
problem.
TheUXproblemisthatendusershavetoconnectallthedotstocoordinatebetween
awidevarietyofdevices,andtointerpretthemeaningofallofthesesensorsto
createpersonalvalue.Formanysimplyconnectedproductsthereissolittleefficiency
tobehadrelativetothecognitiveloadthatit’sjustnotworthit.What’sworse,the
extracognitiveloadisexactlyoppositetowhattheproductpromises,andcustomers
feelintenselydisappointed,perhapsevenbetrayed,whentheyrealizehowlittlethey
getoutofsuchaproductThatmakesmostsuchproductseffectivelyWORSEthan
useless.
Thatpromisegapiswhatdistinguishesagadgetfromatool,whythiseggcartonis
funny,andwhyQuirkywhomadeit,filedforbankruptcyafterburningthrough
hundredsofmillionsofdollars.
15
Howdoyoumakemoneyinthisspace ofdematerializeddevicesandcloudservices?
16
Oneapproachistochangefromanownershipmodeltoasubscriptionmodel.Now
thedevicegivesaccesstoadesiredendresult,withouttheburdensofownershipor
maintenance.TheIoT technologyiswhatgivesanefficientwaytotrackandcharge
forassets.Carsharing,bikesharing,Uber andAirBNB followthismodel.Youdon’t
useiteveryday,sowhyownit?High-endclothingisgoingthisway.Doyoureally
needtoownthatPradahandbagsoyoucanuseittwiceayear?
17
Hewlett Packard’sprinterdivisionisreallyaninkcompanythatalsomakesink
consumptiondevices.SimilarlyAmazonistryingtocornerthemarketonall
consumables,whetherthey’redigital…
18
..orphysical.Their Dashreplenishmentservicecanturnanydevicewith
consumables…
19
…intoanautomaticAmazonreorderingmachine.
TheDashbuttonisanetworkedcomputerwhoseonlypurposeistobeanavatarfor
productswhereit’snotyeteconomicallyfeasibletoincludeconnectedelectronics,
likeamacaroniandcheesebox.That’sgoingtochangeastheelectronicsgetcheaper.
Moreover,thebuttonisasensorforpeople’sintent,whichthendovetailsintothe
realbusinessmodel,whichisnotjustshippingyoumintswhenyou’retoolazyto
leavethehouse…buttoidentifyyourbuyingpatterns,yourcravings,yourimpulses,
sothattheycanpredictthemandshipyoumintsnotwhenyouaskforthem,but
whenyouwantthem.
20
Ithink therealvalueconnectedservicesofferistheirabilitytomakesenseofthe
worldonourbehalf,toreducecognitiveloadbyenablingpeopletointeractwith
devicesatahigherlevelthansimpletelemetry,atthelevelofintentionsandgoals,
ratherthandataandcontrol.Humansarenotbuilttocollectandmakesenseofhuge
amountsofdataacrossmanydevices,ortoarticulateourneedsassystemsof
mutuallyinterdependentcomponents.Computersaregreatatit.
21
Theinterestingthingisthatthisnotjusttheory.
Predictionandresponseisattheheartofthevaluepropositionmanyofthemost
compellingIoT services,startingwiththeNest.TheNestsaysthatitknowsyou.How
doesitknowyou?Itpredictswhatyou’regoingtowantbasedonyourpastbehavior.
22
Amazon’sEchospeaker saysit’scontinuallylearning.Howisthat?Predictivemachine
learningbasedonyouractionsandyourwords.
23
The Birdi smartsmokealarmsaysitwilllearnovertime,whichisagainthesame
thing.
24
Jaguar, learning…ANDintelligent.
25
TheEdyn plantwateringsystemadapts toeverychange.Whatisthatadaptation?
Predictivemachinelearning.
26
Canary,ahomesecurity service.
27
Cocoon,anotherhomesecuritysystem knows.Howdoesitknow?Machinelearning.
28
Here’sfoobot,anairqualityservice.
[Ialsolikehowoneof itsimplicitservicepromisesistoidentify whenyourkidsare
smokingpot.]
29
Silk’sSenseadapts
30
Mistbox sprayswaterintoyourairconditionertoreduceyourenergybill.You’dthink
that’saprettysimpleprocess,butno,it’salwayslearning.
31
Anumberofcompaniesaremakingchipsthatmakemachinelearningmuchcheaper
andmorepower-efficient,whichmeansthatit’sgoingtobeveryeasytoinstallitin
everydevice,fromstreetlightstomedicalequipmenttotoys.It’snotjustlikely,it’s
inevitable.Here’sonethatwasannouncedacoupleofweeksago.
32
33
Theydothisthroughprocessesthathavemanynames,butI’lllumpthemallunderMachine
Learning,whichisabigpartofwhatusedtobecalledArtificialIntelligence.Manyofthecore
ideasheregobacktothe1950sandit’sthebasisofeveryemailspamfilter,soifyou’vehad
yourspamautomaticallyfiltered,you’veexperiencedthevalueofmachinelearning.
AbigpartofMachineLearningispatternrecognition.Wehumansevolvedverysophisticated
facultiestorapidlyidentifyvisualimagesinallkindsofdifficultconditions.Youlookata
pictureofanorangeonaredplateandyoucantellinstantlythatit’snotasunset,butuntil
recentlythatwasreally,reallyhardforacomputer.BecauseofacombinationofMoore’s
Lawandsomebreakthroughs,computershavegottenmuchbetteratpatternrecognitionin
thelastcoupleofyears.
Foracomputer,recognizingsomethingstartswithaprocesswheresomebasicattributesof
animageareextracted,suchastheshapeofboundariesbetweenclustersofpixels,orthe
dominantcolorofapatchofanimage.Thesearecalledfeaturesinmachinelearning.By
examininglotsandlotsofexamplesoffeaturesinanimage,amachinelearningsystembuilds
astatisticalmodelofwhatthatclusterrepresents.
Basicformsofthiskindofimagerecognitionhasbeenusedindustriallyfordecade.Legohas
acompletelyautomatedfactorythatinjectionmoldsamillionLegobricksanhour,examines
everysinglepiece,automaticallysorts,bagsandboxesthem,allusingcomputervision.That’s
relativelyold.
Imagesfrom:Region-basedConvolutionalNetworksforAccurateObjectDetectionand
SemanticSegmentation,R.Girshick,J.Donahue,T.Darrell,J.Malik,IEEETransactionson
PatternAnalysisandMachineIntelligence
Real-TimeImageandVideoProcessing:FromResearchtoRealitybyKehtarnavaz and
Gemadia
34
What’snewisaclassofsystemsthatunderstandthecontentofimages.Theydon’tjustlook
atfeatures,butclustersoffeatures,andclustersofclustersoffeatures,andtheycannow
identifyanorangefromthesettingsun,orapersonfromanairplane,orapolarbearfroma
dalmatian.
ThisiswhyFacebookasksyoutosaywhoisinanimage.It’snotjustforyou,it’sfortheirface
recognizer.
Nowhere’stheinterestingpart:we’rebuilttoidentifypatternsinvisualphenomena,but
we’reprettybadatidentifyingtheminotherkindsofsituations.Forexample,ifyou’veever
triedtounderstandsomeone’sfoodsensitivities,it’sreallyhardtoextractwhatthatperson
isreactingto,evenifyoukeepverycarefultrackofwhatthey’veeaten.We’rejustnotbuilt
forit.Itwasneverevolutionarilysufficientlyimportant,sowedidn’tevolveanorganforit.
Computers,ontheotherhand,don’tcare,andnowthatwe’vefoundreallygoodwaystofind
patternsinvisualimages,thesesametechniquescanfindpatternsinanything.
Insteadofamatrixofpixels,whatifyouhadamatrixofmedicalprescriptions,witheachrow
asthehistoryofoneperson’sprescriptionsfromthefirsttimethatpersonwenttothedoctor
foraproblem,throughwhentheywereprescribedcertainthings,towhentheygotbetter,or
theydidn’t.Thesamekindofsystemcouldlearnthetypicalpatternforprescribing,say,a
wheelchair.Itwouldessentiallyseethegeneralshapeofthesequencefortheprescriptionof
achairovertimeandacrossmanypeople.
Thenifyousawawheelchairbeingprescribedthatwasoutsideofthetypicalpattern,you
couldidentifyit.That’scalledanomalydetection.That’sinfactexactlyhowwebuiltasystem
toidentifyMedicarefraud.Peopleareterribleatthatstuff,butcomputersaregreat.
35
Whenoneofthedimensionsistimeandanotheristheoutcomeofaseriesofactions
youcanmakeapatternrecognizerthatassociatesasequenceofactionswithasetof
statisticalprobabilitiesforpossibleoutcomesbasedondatacollectedacrossawide
varietyofsimilarsituations.Inotherwords,becausepeopleandmachinesbehavein
fairlyconsistentways,thesemachinelearningsystemscanincreasinglypredictthe
futureandattempttoadaptthecurrentsituationtocreateamoredesirable
outcome.
36
37
Asinterestingastheseissuesare,Ithinkthat, moreimportantly,whattheyrepresent
isthatwe’reentering intoanewrelationshipwithourdeviceecosystem,asea
changeinourrelationshiptothebuiltworld.
38
Thinkofasewingmachine.It’sverycomplex,butitstillonlyactsinresponsetous.
39
Computersacting autonomously erodethissimpletool/userrelationship.Predictive
IoTismorethanjustrecommendinganewsong,it’sactingonyourbehalfonthe
basisofitsassumptionaboutwhatyouwant,andwhat’sbestforyou.
Atthedawnofcomputinginthelate1940scyberneticistslikeNorbert Wiener
philosophizedabouttheincreasinglycomplexrelationshipbetweenpeopleand
computers,andhowitwasfundamentallydifferentthanthewayweinteractwith
otherkindsofmachines.Developersworkinginsupervisorycontrolofmanufacturing
machinesandroboticshavehadtodealwiththesequestionspragmaticallyforabout
30years,butthankstotheInternetofThings,thisisnowaproblemthateveryone
willhavetograpplewithgoingforward.
Here’sadiagrambythegreatsTomSheridanandBillVerplank from1978,inwhich
theyillustratefourwaysthatsemi-autonomouscomputersandhumanscanwork
togethertosolveaproblem.
40
By2000SheridanexpandedtheseideaswithParasuraman andWickens todefinea
spectrumofresponsibilitybetweenpeopleandcomputers.Itrangesfromhumans
doingallthework(thisisyouwritinganessay)tocomputersdoingallthework
completelyautonomously(thisisyourcar’sfuelinjectioncontroller).Ofcoursethe
goalistogetasystemtolevel9or10.That’sthemaximumreductionincognitive
load.However,forasystemtoqualifyforthat,ithastobeverystable,itseffects
needtobehighlypredictableand,equallyimportantly,it’sroleneedstobe
adequatelyembeddedinsociety.ItneedstobeOKforacomputertotakeonthat
levelofresponsibility.Attheairportwetrustthemonorailcomputerstowork
withouthumanintervention,butwedon’ttrusttheplaneautopilottodothat,even
though-–asIunderstandit—planescanbasicallyflythemselvesthesedays.
PredictiveIoT devicesgenerallyfallbetween5and7onthisscalerightnow.The
problemisthatthisistheexactrangewhereyou’remaximizingsomeone’scognitive
load,butnotnecessarilydoingalltheworkforthem,sotheresultoftheautomation
hadbetterbeworthit.Thisfundamentallyundermineswhatweexpectfromour
tools,andwhenthattoolistryingtoanticipatewhatwe’retryingtodo,it
fundamentallychangesourworkingrelationshipwithit.
41
Theidealscenariothesethingspaintisprettyseductive.Imagineaworldofespresso
machinesthatstart brewingasyou’rethinkingit’sagoodtimeforcoffee;officelights
thatdimwhenit’ssunnytosaveenergy,andmacandcheesethatneverrunsout.The
problemisthatalthoughthevaluepropositionisofabetteruserexperience,it’s
unspecificinthedetails.Previousmachinelearningsystemswereusedinareassuch
aspredictivemaintenance andfinance.Theyweremadebyandforspecialists.Now
thatthesesystemsareforgeneralconsumers,wehavesomesignificantquestions.
Howexactlyhowwillourexperienceoftheworld,ourabilitytouseallthecollecteddata,becomemoreefficientandmorepleasurable?
We’restillearlyinourunderstandingofpredictivedevices,andinthedisciplineof
whatAaronShapiroofHugehasdubbedAnticipatoryDesign,sorightnowthe
problemsareworsethansolutions.IwanttostartbyarticulatingtheissuesI’ve
observedinourwork.
42
We’veneverhad mechanicalthingsthatmakesignificantdecisionsontheirown.As
devicesadapttheirbehavior,howwilltheycommunicatethatthey’redoingso?Do
westickasignonthemthatsays“adapting”,likethelightonavideocamerasays
“recording”?Shouldmychairvibratewhenadjustingtomyposture?Howwillusers,
orjustpassers-by,knowwhichthingsadapt?Icouldendupsittinguncomfortablefor
alongtime,waitingformychairtochange,beforerealizingitdoesn’tadaptonits
own.Howshouldsmartdevicessettheexpectationthattheymaybehavedifferently
inwhatappearstobeidenticalcircumstances?
How doweknowHOWintelligentthesedevicesare?Peoplealreadyoftenproject
moresmartsondevicesthanthosedevicesactuallyhave,soacoupleofaccurate
predictionsmayimplyamuchbettermodelthanactuallyexists.Howdoweknow
we’renotjust homesteadingtheuncannyvalleyhere?
43
Theironyinpredictivesystemsisthat they’reprettyunpredictable,atleastatfirst.
Whenmachinelearningsystemsarenew,they’reofteninaccurateandunpredictable,
whichisnotwhatweexpectfromourdigitaldevices.60%-70% accuracyistypicalfor
afirstpass,buteven90%accuracyisn’tenoughforapredictivesystemtofeelright,
sinceifit’smakingdecisionsallthetime,it’sgoingtobemakingmistakesallthetime,
too.It’sfineifyourhouseisacoupleofdegreescoolerthanyou’dlike,butwhatif
yourwheelchairrefusestogotoadrinkingfountainnexttoadoorbecauseit’sbeen
trainedondoorsanditcan’ttellthat’snotwhatyoumeaninthisoneinstance?For
allthetimesasystemgetsitright,it’sonthemistakesthatwejudgeitandacouple
suchinstancescanshatterpeople’sconfidence.Anxietyisakindofcognitiveload,
andalittledoubtaboutwhetherasystemisgoingtodotherightthingisenoughto
turnaUXthat’srightmostofthetimeintoonethat’smoretroublethanit’sworth.
Whenthathappens,you’vemorethanlikelylostyourcustomer.
Unfortunately,soonerthanwethink, suchinaccuratepredictivebehaviorisn’tgoing
tobeanisolatedincident.Soonwe’regoingtohave100connecteddevices
simultaneouslyactingonpredictionsaboutus.Ifeachis99%accurate,thenoneis
alwayswrong.Sotheproblem is:Howcanyoudesignauserexperiencetomakea
devicestillfunctional,stillvaluable,stillfun,evenwhenit’sspewingjunkbehavior?
Howcanyoudesignforuncertainty?
Photo CCBY2.0photo2011PopCultureGeektakenbyDougKline:
https://www.flickr.com/photos/popculturegeek/6300931073/
44
Thelastissuecomesasaresult oftheprevioustwo:control.Howcanwemaintain
somelevelofcontroloverthese devices,whentheirbehaviorisbydefinition
statisticalandunpredictable?
Ontheonehandyoucanmangleyourdevice’spredictivebehaviorbygivingittoo
muchdata.WhenIvisitedNestoncetheytoldmethatnoneoftheNestsintheir
officeworkedwellbecausethey’reconstantlyfiddlingwiththem.Inmachinelearning
thisiscalledovertraining.Theotherhand,ifIhavenodirectwaytocontrolitother
thanthroughmyownbehavior,howdoIadjustit?AmazonandNetflix’s
recommendationsystems,whichisakindofpredictiveanalyticssystem,giveyou
somecontextaboutwhytheyrecommendedsomething,butwhatdoIdowhenmy
onlyinterfaceisagardenhose?
45
Hereare7 patternsI’veobservedindevelopingpredictivesystemsthatIthinkmapto
theIoT.FormostoftheseI’mgoingtobeusingexamplesfromNestand
recommendersystemslikeAmazon’s,Google’sandNetflix’s.Recommendersystems
havebeenaroundformorethanadecadeandthey’vebeenextensivelystudied.The
moveintopredictivebehaviorisbuiltonacombinationofrecommendersystemsand
supervisorycontrol,soIrecommendnotreinventingthewheel,butlearningfrom
thosedisciplines.
46
To buildaneffectiveanticipatorymachinelearningsystem,youneedtoknowwhatto
anticipate,andtodothatyouneedtomakeamodelofwhatpeopleneed,valueand
desire.Simplyautomatingexistingactivitieswithoutunderstandingwhypeopledo
them,whattheirgoalsareindoingthem,missesthepointofcreatingvalue.
Predictabilityisveryvaluable,evenwhenthepredictabilityisinsomethingthat’s
flawed.Whenweincludeanticipatorybehaviorinanexperience,we’reessentially
tradingawayanincrediblyvaluablecommoditysothattradehadbetterbeworthit.
Toknowwhetherit’sworthit,weneedtohaveamodelofwhatpeoplevaluewhich
we’rereplacingoraugmenting.
47
Whatgoes intothatmentalmodel?
Therearelotsofwaystostructurehowyourepresentpeople’sviewoftheworld.It’s
asignificantfocusofcognitivescience,andIcan’tdoitjustice,buthere’sanicelistI
grabbedfromtheintelligentagentliterature.
Asadesigner,manyoftheseboildowntodecisions.Whatdecisionwillan
anticipatorysystemhelpsomeonemake?Whatdecisionswillitmakeonthat
person’sbehalf?Whataretheparametersofthatdecision?Forexample,ifIhada
real-timebloodglucosemonitorandinsulinpumpthatadjustedmybloodglucosein
realtime,whichofmydecisionswoulditmakeforme?Whichdecisionswouldittell
mehowtomake?Whichdecisionswoulditgivemeadviceabout?
Withoutaclearclearlyarticulatedstoryaboutwhatdecisionsasystemhelps
someonemake,Ibelieveyoudon’thaveaclearstoryaboutwhatvalueitbrings
them.Howdoyoufigureoutwhatthosedecisionsare?Youtalktopeople.User
research.Ethnography.Leavingtheoffice.
48
Oneofthegreatcliches inUXdesignisthesearchfordelight,suchastheseasonally
changingbackgroundsinGoogleCalendar.Mydefinitionfordelightisthatit’s
functionalitythatsubvertspeople’snear-termexpectations,butsupportstheirlong-
termneedsanddesires.Thisisparticularlyimportantindesigningpredictivesystems,
becauseifyousubvertexpectationsWITHOUTsupportingtheirneeds,youget
cognitivedissonanceandyouhaveviolatedtheirmentalmodel.
49
Becausemachinemeansyour toolsadapttoyouandlearnsfromyou,adaptivetoolsare
morelikeapprentices,ratherthanimplementsandouruseofthemismorelikea
conversationratherthanthanlineartooluse.Infact,IheardoneofNest’sUXdesignerssay
thatheconsideredusers’evolvingrelationshipstotheNestasaconversation.
Thisisespeciallyrelevantintheeraofchatbots andvoiceUI.Ifyoulistentoahuman
conversation,it’salmostneveralinear,straightforward,well-structuredprocess.Westop,
werephrase,weaskforcorrections,wetalkpasteachother,weinterrupt.Morelikelythan
not,thisishowapredictivemachinelearningsystemwillinteractwithpeople,fromwhomit
willwantguidance,confirmation,andwhowillaskitforrecommendationsorchangestoits
behavior.
Ethnomethdologists andconversationanalystshavebeenmodelinghowpeopletalktoeach
otherforabout40years,soI’mgoingtoborrowsomeoftheirconcepts.
• Sequenceorganizationisaboutorganizingactionintime.Whathappensfirst,what
happensnext?Howdothetwopartiesexpandonambiguity?Forexample,ifahome
securitysystemdecidesyou’renothome,itcantellyou“Iseeyou’redrivingawayfrom
home.I’mgoingtoturnallthealarmson.”Youcanthensay“Alloftheexceptfortheback
yard.”
• Turn-takingiscritical.Wedon’tjustsimplytaketurnswhentalking,wecontinuously
providefeedbackandcorrect.Wehaveexpectationsforwhoseturnisnextandwhat
they’resupposedtodo.“Ok,chair,I’msittinghere,nowit’syourturn.Confirmyouknow
I’mhere.Warnmeifyou’regoingtoadjust.”
• Repairisbacktracking,clarifying,continuingafteraninterruption,etc.Whathappens
whentheexpectedsequence,eitherfromtheperspectiveofthepersonortheservice,is
brokenandneedstobereconstructed?
50
Inadditiontoteachingapprentices aboutourneeds,wealsolearnfromapprentices
whattheircapabilitiesareandwhytheymadecertaindecisions,ratherthanothers,
whendoingthethingswetaughtthemtodo.Thisisbothapartofhowtheylearn
aboutusandhowwelearntoworkwiththemeffectively.TheBMWiDrive system
wasnotoriousforitsUI,whichdidn’ttellyouwhatitcouldorcouldn’tdo,andhowto
doit.Youhadaknobandthatwasbasicallyit.
HowdoIinterrogateanadaptivesystemtounderstandwhatitcando,andtoaskit
toexplainwhatitjustdid.
HowdoyouknowwhatSiriorGoogleNowhavelearnedtodo?Well,youusethe
app.Butwhataboutservicesforwhichyoudon’thaveadisplay?Chatbots todayare
essentiallycommandlineinterfaces.Theyknowspecificwordsandsequences,but
whatifthosecommandschangeovertime?Whatifthedevicelearnsnewthingsover
time?
51
Thenextpatternisthatyouneedauserstoryforeverystageofthemachinelearning
andpredictionprocess,evenforstepsthatseemsinvisible.Howwillyouincentivize
peopletoaddtheirbehaviordatatothesystematall?WhyshouldIuploadmycar’s
dashcam videotoyourtrafficpredictionsystemEVERYDAY?Howwillyou
communicateyou’reextractingfeatures?IlikethewaythatGooglespeechtotext
showsyoupartialphrasesasyou’respeakingintoit,andhowitcorrectsitself.That
smallbitoffeedbacktellspeopleit’spullinginformationoutandittrainsusershow
tomeetthealgorithmhalfway.Howdomachine-generatedclassificationscompareto
people’sorganizationofthesamephenomena?Howisacontextmodelpresentedto
endusersanddevelopers?Howwillyougetpeopletotrainitandtellyouwhenthe
modeliswrong?Doesthefinalbehavioractuallymatchtheirexpectation?
52
Sincepredictivesystemsareneitherconsistent, norarethereasonsfortheirbehavior
clear,thiscanbereallyconfusing.Thesamethingcanbehavedifferentlyinwhat
appeartobesimilarcircumstances.Ifweunderminepeople’sconfidenceinasystem
byviolatingtheirexpectations,they’relikelytobedisappointedandstopusingit.
Whenwe’redealingwithahuman orananimal,unpredictablebehaviorsare
expectedandtolerated,butthat’snotthecasewithcomputers.ApredictiveUX
needstodoistosetpeople’sexpectationsappropriately.Itneedstoexplain the
natureofthedevice, todescribeitistryingtopredict, thatit’stryingtoadapt,that
it’sgoingtosometimesbewrong,toexplainhowit’slearning,andhowlongit’lltake
beforeitcrossesoverfromcreatingmoretroublethanbenefit.
Recommendersystems,suchasGoogleNow,describewhyacertainkindofcontent
wasselected,andthatsetstheexpectationthatinthefuturethesystemwill
recommendotherthingsbasedonotherkindsofcontentyou’verequested.Nest’s
FAQkindofburiestheinformation,butitdoesexplainthatyoushouldn’texpectyour
thermostattomakeamodelofwhenyou’rehomeornotuntilit’sbeenoperatingfor
aweekorso.
53
Aboutten yearsagoTimo Arnall andhisstudentstriedtoaddressasimilarsetof
questionsaroundinteractionswithRFID-enableddevicesbycreatinganiconography
systemthatcommunicatedtopotentialusersthatthesedeviceshadfunctionality
thatwasinvisiblefromtheoutside.Perhapsweneedsomethinglikethisforbehavior
createdbypredictivebehavior?
54
Predictivebehavior,isallabouttime,aboutsequencesofactivities.Manypredictive
UXissues aroundexpectationsanduncertaintyhavetimeastheirbasis:whatwere
youexpectingtohappenandwhy.Ifitdidn’thappen,why?Ifsomethingelse
happened,orithappenedatanunexpectedtime,whydidthathappen?
Knowing thatadevicehasactedonyourbehalf,andthatit’sgoingtoact—andHOW
it’sgoingtoact—inthefutureisimportanttogivingpeopleamodelofhowit’s
working,settingtheirexpectations,reducingtheuncertainty.Nest,forexample,hasa
calendarofitsexpectedbehavior,anditshowsthatit’sactingonyourbehalfto
changethetemperature,andwhenyoucanexpectthattemperaturewillbereached.
55
Youhavetogivepeopleaclear waytoteachthesystemandtellitwhenitsmodelis
wrong.Statisticalsystems,bydefinition,don’thavesimplerulesthatcanbechanged.
Therearen’tobvious handlestoturnordialstoadjust,becauseeverythingis
probabilistic.Ifthemodelismadefromdatacollectedbyseveraldevices,which
deviceshouldIinteractwithtogetittochangeitsbehavior?GoogleNowasks
whetherIwantmoreinformationfromasiteIvisited,Amazon showsaexplanation
ofwhyitgavemeasuggestion.MappingthistotheconsumerIoT meanswaymore
explanationthanwe’recurrentlygetting,whichiseitherthatathinghashappened,
orithasn’t.
56
Finally,don’tautomate. Thesesystemsshouldn’ttrytoreplacepeople,buttosupport
them,toaugment andextenttheircapabilities,tohelpthembebetteratwhatthey
wanttodo,nottoreplacethem.
Forexample,EmberfromMeshfire, isamachinelearningassistantforsocialmedia
management.Itdoesn’ttrytoreplacethesocialmediamanager.Insteaditmanages
themediamanager’stodo list.Itaddsthingsthatitthinksaregoingtobeinteresting,
deletesoldthings,andreprioritizesthemanager’slistbasedonwhatitthinksis
important.Ithinkthisisagoodmodelforhowsuchsystemscanaddvaluetoa
person’sexperiencewithoutcreatingasituationwhererandom,unexplained
behaviorsconfusepeople,frustratethemandmakethemfeelpowerless.Emberisan
augmentationtothesocialmediamanager,ithelpsthatpersonfocusonwhat’s
importantsothattheycanbesmarterabouttheirdecisions.Itdoesn’ttrytobe
smarterthantheyare.HowcanourdevicesHELPus,ratherthantryingtoreplaceus?
57
Finally,anantipattern:makingpeople doallofthetraining,askingthemtoidentify
whetherabehaviorisappropriateornot,shouldbedoneselectivelyand
infrequently.Yes,itwillreallyhelpyoursupervisedmodel’saccuracytohavepeople
identifythecorrectpositivesfromthefalsepositives,butunlessyou’repayingthese
people,it’sincrediblyannoyingtohavecustomersdoitallthetime.LastFridayone
consumerIoT productwithamachinelearningsystemI’mplayingwithaskedmeto
classifyitsoutputat1:11PM,thenagainat1:26,andagainat1:47andagainand
again.Ithinkitwasonroughlyten-minutesensingcycle,andateverycycleittriedto
makeadecision,andaskedmetoverifyit.I’msureit’sstilldoingit,butIturnedoff
allnotificationsfromit,andnowI’mconsideringturningitoffentirely.Peoplewill
sometimeswillinglyactassensorsandactuatorsforyoursystem,butbecausethey
arenotmachines,theywillnotdoitallthetimeandyou’rejustgoingtohavetofind
abetterwaytotrainyourmodel.
58
Finally,formetheIoT isnotaboutthethings,buttheexperiencecreatedby the
servicesforwhichthethings areavatars.
59
Ultimatelyweareusingthesetoolstoextendourcapabilities,tousethedigitalworld
asanextensionofourminds.Todothatwellwehavetorespectthatasinteresting
andpowerfulasthesetechnologiesare,theyarestillintheirinfancy,andourjobas
entrepreneurs,developersanddesignerswillbetocreatesystems,services,thathelp
people,ratherthanaddingextraworkinthenameofsimplisticautomation.Whatwe
wanttocreateisasymbioticrelationshipwherewe,andourpredictivesystems,work
togethertocreateaworldthatprovidesthemostvalue,fortheleastcost,forthe
mostpeople,forthelongesttime.
Wearecurrentlyshovelingourolddevicesintothisnewmedium.Wehavenotyet
figuredoutwhattheessentialcapabilitiesofthisnewmediumare.
LiteralMcLuhanquotation:"Thecontentofthepressisliterarystatement,asthe
contentofthebookisspeech,andthecontentofthemovieisthenovel."
60
Thankyou.
61