ismar 2014 ar workshop: designing location based experiences
DESCRIPTION
My presentation from the 2014 ISMAR workshop series. Location-based experience design workshop is focused on exploring and teaching about the intersection between story, location, and technology for augmented and mixed reality products and installation development. Here I focused on combining and drawing links between UX tools such as user journey maps, as a design tool for location experiences from the user perspective. This influences game mechanics, world building, etc. to form the location experience. As use cases I presented two projects I have been involved with, Ghosts of Venice and Lost In Reality.TRANSCRIPT
Location Based Experience Design Workshop
Mark Melnykowycz
ISMAR 2014 Sept. 9th 2014
1
Mark Melnykowycz [email protected]
Story Structure
Story Location
User Experience
A/MR Technology
Location
Hero’s Journey Changing
structures
Books, movies, ebooks, games
IxD / UXFundamentals
of IxD
Prototyping
Consumption Patterns
User Journey
Game Mechanics Metaio
SDK
Tracking / Positioning
AR / MR Fundamental differences
Technology
Teaching Users How
to Play
World Building
Game Space Rules
2014 ISMAR Workshop:Location based experience design
Lost In RealityMobile storytelling
Ghosts of VeniceMixed-reality film
2
Mark Melnykowycz [email protected] 4
Generic Location App Use Case
User goes to a location
Content or interaction is
triggered
Mark Melnykowycz [email protected] 5
Designing Location Experiences
Why is the person in that
location?
How are they interacting with the location?
Why are they motivated to engage in this
experience?
How are experiences
planned?
What technologies are available?
Mark Melnykowycz [email protected] 6
Key Physical Elements
LocationPosition /
Orientation
InteractionContent
Mark Melnykowycz [email protected] 7
User motivations / immersion
Sketching out user flows / prototyping / product definition
Building interaction / defining app world
Technology trends
Communication patterns
Mark Melnykowycz [email protected] 8
IoT/M2M Devices
Device Fragmentation Evolving Communication
Tech and Society Trends
Mark Melnykowycz [email protected] 10
50 Billion!By 2020*!
Quantified User!
Quantified Self!
IoT!
http://share.cisco.com/internet-of-things.html
Connected City
Location Interaction
Large connected ecosystem
Connected Device Evolution
Mark Melnykowycz [email protected] 11
Location
Time
Heart Rate
Sensor Fusion User State
Quantified User
Mark Melnykowycz [email protected] 12
Defined UX in Closed SystemsD
evic
eIn
tera
ctio
n
Interaction and consumption patterns are learned making the transition to new devices easy
Mark Melnykowycz [email protected] 13
Ubiquitous C
omputing
Defined UX Space
Distributed UX
Fragmented UX Patterns
Mark Melnykowycz [email protected] 14
How are users trained to interact with locations?
Maps
Location as a Device Experience
Mark Melnykowycz [email protected] 15
Determine orientation in location using navigation aids
Maps
Identify physical anchor points to define position
Use technology to identify information
around you
Future actions based on understanding where you are
and what you can explore
Mark Melnykowycz [email protected] 17
Short Evolution of Communication
Face to Face
Web
Email Static HTML Blogging Instant Messaging
Social Networks
Location ServicesWriting
PrintingCopier
TelephoneMobile SMS
MMS
Fragmented Lines!Push/Pull!
Transmedia
Mark Melnykowycz [email protected]
Content Container Consumption Velocity
The User
User Experience Design
18
Consumption Concepts
Mark Melnykowycz [email protected] 19
Content
The User
Information transferred between containers and consumed by people
Consuming Media
Mark Melnykowycz [email protected] 20
Physical Container
The User
A real, tangible thing used to hold, and through which content can be consumed
Has been designed, has structure and fills up physical space
Mark Melnykowycz [email protected] 21
The User
A virtual thing used to hold, and through which content can be consumed
Has structure, fills up space and can scale
Defined through graphic design and UI engineering
Virtual Container
Mark Melnykowycz [email protected] 22
Consumption
The User
Understand/Retain information Make it part of your knowledge base How fast is this?
Velocity
Content
Vocabulary
Mark Melnykowycz [email protected] 23
Landing Page GridContent Home
Print Web Mobile
Pull out bits of information
Scan - look for anchors
Save an idea and move on
Linear progression
Consumption Patterns
Mark Melnykowycz [email protected] 25
Digital Consumption
App/Webpage Social Media Video Augmented Reality
Audience
CuratorPush/Pull
Mark Melnykowycz [email protected] 26
SnowFall
New York Times / http://www.nytimes.com/projects/2012/snow-fall/
Scroll page, easy to pull out bits of the story and be immersed in graphics, story, characters.
Mark Melnykowycz [email protected] 27
Text, animations, videos, different themes, consistent narrative
Pull-based consumption and reading of content
Mark Melnykowycz [email protected] 28
Parallax, easy to explore Pull consumption Easy to design/prototype
https://www.scrollkit.com
Mark Melnykowycz [email protected] 29
Easy to integrate, drag/drop images, embedded media (video/audio/iframes)
Custom URL or publish on scrollkit.com
Mark Melnykowycz [email protected] 30
Social Media Curation
More powerful than Tumblr, connect to and search social media channels, present content in sequence
Flow/steam Slideshow
Mark Melnykowycz [email protected] 31
Connect to social media channels
Arrange elements to group together Add text and links
Mark Melnykowycz [email protected] 32
Publish as landing page on BrickFlow, share link to social media channels, remix
Embed ex te r na l l y as iframe element, view as slideshow experience
Mark Melnykowycz [email protected] 33
Connect to and search social media channelsAdd content, text, etc.
Mark Melnykowycz [email protected] 34
Mixed Video/Interaction
Non-linear video navigation Clickable elements Easy assembly
Combine multi-media elements View in video/slide format Videos, audio, animations, text, etc.
Mark Melnykowycz [email protected] 35
http://www.raptmedia.com/product
Add clickable elements to videos, seamlessly jump from one video to another.
Allow choices for jumping between storylines/elements
Mark Melnykowycz [email protected] 36
Interactive layer allows clicking/hyperlinks
Different video clips can be linked together
User can make decisions at plot points
Easy to create alternative story lines or allow exploration
Mark Melnykowycz [email protected] 37
Connect to channels, ability to upload
Arrange elements Share
www.zeega.com
Mark Melnykowycz [email protected]
Empathy Define Ideation Prototype Test
Discovery High-Level Solution Product Solution
Product/Experience Design PhasesUser Needs
Focu
s / F
lare
App Prototype
39
Experience Planning Interactivity Prototyping
Story/Location Technology
Mark Melnykowycz [email protected] 40
Story Consumption Container
What is the story, the world it takes place in, the character, etc.
How will this story world be consumed by the audience or users?
How will the consumption be realized, physically/virtually designed, etc?
Know your User!Wireframe/prototyping Webpage/App Video/ebook
Know your Audience!Story boarding Pre-Visualization
User Experience/Interaction
Mark Melnykowycz [email protected] 41
Creative Product
Creates stories, art, movies, etc.
Builds physical/virtual products
The UsersThe Audience
User Experience/Interaction
Find your audience
Prototype/Test/Explore
Mark Melnykowycz [email protected] 42
How will the consumption be realized, physically/virtually designed, etc?
Wireframe/prototyping Webpage
Video ebook
ContainerConsumption
User Experience/Interaction
User Journey
Paper/Digital Mockup
Interactive Prototype
Final Experience
Mark Melnykowycz [email protected] 43
How does the user get into your story and engage in your experience?
User Journey
User sees meme via Facebook/Tumblr
Searches for Zurich Dada
Game on Google
Starts reading/learning about the Dada ghost
in Zurich
Need to have good SEO base on landing page
Need fast consumption on desktop/mobile
Parallax page design
User is push/pull fragmented reader
Mark Melnykowycz [email protected] 44
Design representation of your app/container to communicate and test the user experience
Paper/Digital Mockup/Prototype
Communicate between design / development
Communicate with potential partners / investors / users
Testing tool for fast user feedback before development
Pape
rM
ocku
pD
emo
App
Mark Melnykowycz [email protected] 45
Web/App Prototyping
Keynote Powerpoint OmniGraffle GUIToolkits
HotGloo JetStrap
AppCooker BluePrint iMockups
Stencils
Visio Balsamiq
Axure iRise MX
Photoshop Illustrator
iPad
OnlineDesktop
Mark Melnykowycz [email protected] 47
Narrative
•Story, a recounting of a sequence of events* •An account of imaginary or real people and events
told for entertainment: "an adventure story"**
•A narrative (or story) is any account that presents connected events***
•A spoken or written account of connected events; a story: "a bare narrative of the details"**
*Wikipedia **Google dictionary ***Oxford English Dictionary (online)
Story
Mark Melnykowycz [email protected] 49
...storytelling is a vehicle used to combine emotions and information
...and communicate those feelings and information in a structure which gives us comfort or instills feelings of fear,
optimism, etc.
Shared emotions help form the fabric of society
Mark Melnykowycz [email protected] 52
Birth Death
Physical time is linear Emotional memory is non-linear
Mark Melnykowycz [email protected] 53
Connect current events to good feelings from the past
Play out emotions we can’t explore in reality
Hope in the future and our confidence to meet challenges
Story
Mark Melnykowycz [email protected] 54
...makes me feel good about my chances of surviving a dramatic terrorist attack
...statistically it will probably never happen, but I’m sort of mentally prepared if it does
Die Hard, 20th Century Fox
Die Hard
Mark Melnykowycz [email protected]
What is traditional story structure? Does it conflict with our fragmented communication
patterns?
55
Story Structure
Many different models of story structure for different media
56
Mark Melnykowycz [email protected] 59
TransformationChallenge
Over-coming extreme odds...to succeed in the end…
Mark Melnykowycz [email protected]
http://www.musik-therapie.at/PederHill/Structure&Plot.htm
60
http://wandertones.deviantart.com/art/Standard-Story-Structure-178276496
Rising drama leading to an apex with essential plot points and local challenges/resolutions
Linear Story Structure
Mark Melnykowycz [email protected] 61
“Campbell explores the theory that important
myths from around the world which have
survived for thousands of years all share a
fundamental structure, which Campbell called
the monomyth.”
https://en.wikipedia.org/wiki/The_Hero_with_a_Thousand_Faces
Joseph Campbell
Mark Melnykowycz [email protected] 62
THE HERO IS INTRODUCED IN HIS ORDINARY WORLD THE CALL TO ADVENTURE
THE HERO IS RELUCTANT AT FIRST THE HERO IS ENCOURAGED BY THE WISE OLD MAN OR WOMAN
THE HERO PASSES THE FIRST THRESHOLD THE HERO ENCOUNTERS TESTS AND HELPERS
THE HERO REACHES THE INNERMOST CAVE THE HERO ENDURES THE SUPREME ORDEAL
THE HERO SIEZES THE SWORD THE ROAD BACK RESURRECTION
RETURN WITH THE ELIXIR
http://www.cs.uu.nl/docs/vakken/b2go/docs/hero_journey.pdf
Joseph Campbell
Mark Melnykowycz [email protected] 63
The Hero’s Journey repeated in countless media…
Mark Melnykowycz [email protected] 64
Suspension of disbelief or willing suspension of disbelief is a term coined in 1817 by the poet
and aesthetic philosopher Samuel Taylor Coleridge, who suggested that if a writer could infuse a "human interest and a semblance of truth" into a fantastic tale, the reader would
suspend judgement concerning the implausibility of the narrative.
http://en.wikipedia.org/wiki/Suspension_of_disbelief
Suspension of Disbelief
Mark Melnykowycz [email protected] 65
Why is the Hero’s Journey structure so popular in movies and media?
Fairy Tales, myths, Star Wars, The Matrix, etc.
Do we like it because we expect it and are conditioned for it?
Is it inherent in our minds and DNA?
Does it hold us back in life? Must we be chosen to do great things by someone else?
Mark Melnykowycz [email protected] 67
Confront a challenge
Understand your progress
Play
Interactive with the world
Interact with other players
Actions have consequences
Conquer small quests
Games
Mark Melnykowycz [email protected] 68
Story
Players
Game world / field of play
Conflict / ChallengeMechanics
Resources
Game Elements
Mark Melnykowycz [email protected] 69
Players understand the rules
Players overcome challenges
Players understand their place in the
context of the game
Points System
Levels
Time
Reason for the experience to exist
Know how to interact with the experience
Game Mechanics
Game Enjoyment
70
The StoryThe Challenge The Game Play
http
://en
.wik
iped
ia.o
rg/w
iki/D
oom
_(19
93_v
ideo
_gam
e)
http
://z2
.com
/gam
e/m
etal
stor
m-o
nlin
e/
http
://en
.wik
iped
ia.o
rg/w
iki/P
uzzl
e_Bo
bble
http
://en
.wik
iped
ia.o
rg/w
iki/W
orld
_of_
War
craf
t
Mark Melnykowycz [email protected] 72
Character evolves
Character overcomes challenges
Character presented with challenge
Generic HeroGeneric Threat
Finds courage... Triumphs!
A Generic Story
Mark Melnykowycz [email protected] 73
Character evolves
A character evolves/changes due to conflict inherent in the story
Relationships
Environment
Character Change Through Conflict
Mark Melnykowycz [email protected] 74
Is the user a participant or an outside viewer in a location game or experience?
Who is the Main Character?
Mark Melnykowycz [email protected] 75
Story Development
Character Driven World Driven
How will the world affect character development?
Generic environment, used as a setting for character interaction to take place.
The story of a character in a given environment, which forces action / conflict
...become a different person
...character traits stay the same
Mark Melnykowycz [email protected] 76
Contemporary super hero
Traditional action hero
Die Hard Arnold Schwarzenegger
Ironman, Thor, Spiderman, etc.
Watchmen Dark Night
Man of Steel X-men
Simple dialogue, hero never dies, always
triumphs
Question their actions and place in the story, supporting characters
die, like real life.
Good vs. Evil Good? vs. Grey Zone
Mark Melnykowycz [email protected] 77
Location Time Physics
Where is the world? Is it a city, a new planet, a coffee house?
Take a character from another time
or use a world from a different
time?
If it’s a new world, does it respect the physics of Earth as
we understand them?
World
78
X | Media | Lab Switzerland 2013: Transmedia Alex McDowell – Transmedia’s Renaissance Man: Narrative Designer for Minority Report; Fight Club; Man of Steel; Associate Professor, USC – School of Cinematic Arts; Visiting Artist, MIT Media Lab (Hollywood)
79
X | Media | Lab Switzerland 2013: Transmedia Alex McDowell – Transmedia’s Renaissance Man: Narrative Designer for Minority Report; Fight Club; Man of Steel; Associate Professor, USC – School of Cinematic Arts; Visiting Artist, MIT Media Lab (Hollywood)
80
X | Media | Lab Switzerland 2013: Transmedia Alex McDowell – Transmedia’s Renaissance Man: Narrative Designer for Minority Report; Fight Club; Man of Steel; Associate Professor, USC – School of Cinematic Arts; Visiting Artist, MIT Media Lab (Hollywood)
Mark Melnykowycz [email protected] 81
How does a character move around the world?
Character Journey
Character becomes sick Must go to
hospital
How would she die? Never invented
gasoline, all hydrogen
Transportation logistics
Mark Melnykowycz [email protected] 82
Physical Virtual World Building / Prototyping
Unity 3D!• Multi-platform Game
development • Terrain building • Visualization • Integration of buildings,
people, etc.
Mark Melnykowycz [email protected] 83
Sketch out worlds
The Hobbit Maphttp://tolkiengateway.net/wiki/File:Christopher_Tolkien_-_Map_of_Wilderland.jpg
Physical story worlds define logic of story progression
Mark Melnykowycz [email protected] 84
Paint terrain and topography features !Add textures for grass, rock, water, etc. !Add buildings, people, tanks, etc.
Mark Melnykowycz [email protected] 85Load new models
Paint on topography
Load textures
Use different brushes
Unity User Interface
Mark Melnykowycz [email protected]
Generic AR Experience
Marker Marker-less Point-cloud Edge-based
App tracks feature in reality Content positioningUser engaged with
app
Virtual content
87
Mark Melnykowycz [email protected] 89
Depth Cameras
Desktop
LaptopMobile
Stru
ctur
e.io
Proj
ect T
ango
92
Marco Gatti
Film Maker / Director
Sara Mautino
AR Glass Expert
Mark Melnykowycz
UX / AR coding
Visual Effects
Story Tech
Production
ghostsofvenice.it
93
Place Ghosts in the Environment
ghostsofvenice.it
Take users on a ghost story through Venice, mixing story elements and 3D animations into the local environment
Film vs. App
94
C o n s u m e d b y watching/listening while stationary
L o c a t i o n a n d con ten t can be mixed to provide more immers ive experience
ghostsofvenice.it
Augmented Reality
95
Technology used to slightly modify the perception of the environment of the user
GPS location Media overlay
Image recognition
Env i ronment or user triggers new actions in the software
Feature tracking
Image Video Audio Text1
2
3
4
ghostsofvenice.it
Story - Technology Development
96
Mark Melnykowycz [email protected]
Transmedia Narrative
Online Story Hard Copy
Mobile App
Educate users of the background story before starting
Give users something tangible, objects from t h e g h o s t s , t h e i r letters, etc.
Main point of contact between user, story, and location
ghostsofvenice.it97
Mark Melnykowycz [email protected]
Story Development
App Design
App Development
Media Production
How do all of these key elements come together in the transmedia
development process?ghostsofvenice.it98
Mark Melnykowycz [email protected]
Story/App Workflow
Story App Design
App Development
User Experience User Interaction
Background stories of ghostsHow stories are
delivered
Implement UX/UI in App Development
Test/Iterate
Prototype
Production App
ghostsofvenice.it99
Mark Melnykowycz [email protected]
Story/Media CreationStory Locations Ghosts
Choose locations Choose trigger
elements
Background story for each ghost
Visual Design Casting Filming
Post-production
Ready to distribute on
mobile device
Mobile Media
Test App
Code into data model for
viewing/testing
Production App
ghostsofvenice.it100
Mark Melnykowycz [email protected]
Sets the stage, gives tangibility to the story experience, being in the same physical place as the ghosts
App is used to view the ghosts, to connect with the “being” the user has learned about
The impact of seeing the ghosts comes from the emotional connection to the story behind them
How can the story and location be interactive with the participants? This increases the emotional connection to the experience
Transmedia LandscapeA transmedia project integrates a larger narrative across different distribution
channels, we need to develop a plan for how this would be realized
Location
Mobile AppStory Channels
Interaction
Real book/diary of ghosts Wikipedia page each individual Artifacts of ghosts in the city/museums...what else? Establish emotional connection and empathy to ghost characters ghostsofvenice.it101
Augmented Reality Technology Needs
102
Mark Melnykowycz [email protected] 103
User Walks Through Area
Location Aware
Image recognition
Event Playing
User sees they are in the area where
a trigger is
User searches for physical
trigger
Location triggers haptic/vibration on
device
Content plays through device
ghostsofvenice.it
User Flow
ghostsofvenice.it 104
Alpha Mask
Transparent background
Overlay in video feed
Image Recognition for Location?
Software recognizes specific features of the environment
Image pre-programmed in to the dev ice o r queried from a server
Recognition may not be automatic, depends on hardware speed
ghostsofvenice.it 105
Install Junaio
Access GOV channel
View content on image trigger
106ghostsofvenice.it
Changing light conditions will impact
tracking ability
Image target not reliable enough for spatial location use
107ghostsofvenice.it
Feature RecognitionSoftware recognizes specific features of the environment
P o i n t - c l o u d m a p created to aid in feature recognition
CAD or edge-based tracking can be tested as well
ghostsofvenice.it 108
Metaio Edge-based Tracking
CAD model overlay on environment
Better in low-light tracking scenarios
Where do I get building CAD?
109
www.metaio.com
https://dev.metaio.com/sdk/documentation/tracking-config/optical-tracking/cad-model/
App Development Flow
110
Layar/Junaio Metaio SDK Unity Beta App
Test the concept of playing audio o r v i d e o fi l e s based on GPS location with AR Browser
Initial Test Functional Prototype Initial Release
I m p l e m e n t v i d e o overlay concept in a na t i ve iOS /Andro id application, testing with users to understand the UX better
With a developed s t o r y l i n e / p l o t , producing the full m i x e d r e a l i t y exper ience and broad testing with users
ghostsofvenice.it
Mark Melnykowycz [email protected] 111
Use 3D scan of objects to create CAD for model
positioning
Experiment with 3D mapping for
model positioning
Mobile 3D Scanning
Mark Melnykowycz [email protected] 112
Ghosts as model scans
Match to environmental
features
Combine Person and Environment Models
Marco Gatti
Film Maker / Director
Sara Mautino
AR Glass Expert
Mark Melnykowycz
UX / AR Development
Visual Effects
Team
www.ghostsofvenice.it113
Lost In Reality
114
Lost In Reality
Mobile App Web App
Search Consume Create Share
Create Edit
Storytelling Social Network
115lostinreality.net [email protected]
Francisco Campelo José Rodrigues Mark Melnykowycz Ismail Cimen
App DeveloperPartner and
Story Development
Project Lead UX/UI
Design UX/UI
Team
Lisbon, Portugal Zurich, Switzerland
116lostinreality.net [email protected]
Emo8onsLoca8ons
People
How can we use story to add to the information context of locations?
117lostinreality.net [email protected]
Stories saved
For Storytellers
Authors can write loca8on stories on the web app
Users find stories based on loca8on
and tags
Users can write stories on the mobile app
Story server
118lostinreality.net [email protected]
Tourists have more unique experiences in
new ci8es
For Authors
Authors can write loca8on stories on the web app
Users can download stories
Locals have new perspec8ves on
their city
Loca8on used as integral element of
stories
119lostinreality.net [email protected]
Users search and access loca8on-‐stories on
demand with mobile app
For Readers
Story elements triggered based on loca8on in
ci8es
Context of stories gives more value to loca8ons
120lostinreality.net [email protected]
Location Stories vs. Social Media
Location Services
Context?
Social Networks
Location stories give structure and context to information and combats the social media noise
Social media evolved from blogging as faster way to publish ideas
Social media updates are only useful for a short time frame
“Facebook/Twitter shows people by who they are, what they have done and said”
Location services give an idea of what a person can do in a place (irregardless of when)
“Locat ion s tor ies show people the experience they’ve had”
“Location stories let people follow experiences and have their own”
121lostinreality.net [email protected]
Users can create, search, and follow location stories
Focus on location-based storytelling
Emotional engagement to location
Primary Goals of LIR
Smart phone application
Dashboard Story Library
Local Story Search Story Element
Story elements delivered to user at GPS location
(triggered)
122lostinreality.net
Mobile App Flow
125
User can follow different stories in a city as they
walk around.
lostinreality.net [email protected]
Mark Melnykowycz [email protected]
User motivations / immersion
Sketching out user flows / prototyping / product definition
Building interaction / defining app world
Technology trends
Communication patterns
126