meaning modes in design - fluxible 2016
TRANSCRIPT
Invisiblethingsaboutmeaning…
We’re going to take a visual tour of some invisible things about meaning, some properties of meaning, that matter with the kinds of designs we are bringing into the world these days.
TwoJJGs
JessieJamesGarrett JamesJeromeGibson1950s-70s
Image credit: @UXweek Twitter account Image credit: Wikipedia
Before we visualize these properties, I must mention two JJGs. One JJG is Jessie James Garrett, our contemporary in experience design. The other is James Jerome Gibson, the founder of the field of ecological psychology, working half a century ago.
TwoJJGs
JessieJamesGarrett JamesJeromeGibson2002 1950s-70s
In 2002, Jessie James Garrett gave us an exploded planes view of the elements of user experience, in which the structure, skeleton, and surface planes treat how we use information as raw material in terms of its architecture, visualization, and perception.
TwoJJGs
JessieJamesGarrett J.J.Gibson2002 1979
J.J. Gibson gave us a naturalized view of how we perceive information, how we pick it up directly as we act in the environment. (For present work using the ecological framework developed by Gibson, see Golanka S., and Wilson A. (2013). Embodied cognition is not what you think it is, Frontiers in Psychology )
TwoJJGs
JessieJamesGarrett J.J.Gibson
MEANING
And both of them worry about how information influences meaning.
Experienceisimpossibletomodelandfullydesign
Rocket contrail by Seay Photography via Creative Commons
Emergent
IneffableIntangible
Subjective
JessieJamesGarrettClosingPlenary,IASummit2016
In his closing plenary in the 2016 Information Architecture Summit, Jessie James Garrett lamented with us that experience is impossible to model and fully design. It is emergent, subjective, intangible, ineffable.
Shiftingpatternsofawareness
Framesofmeaning
Anunfoldingasahumanengagesadesign JessieJamesGarrettClosingPlenary,IASummit2016
Rocket contrail by Seay Photography via Creative Commons
Experience is an unfolding as a human engages a design that creates shifting patterns of awareness, and frames of meaning that a human assigns to how that particular design in that particular moment-to-moment shifts her patterns of awareness.
Howfarcanwego? JessieJamesGarrettClosingPlenary,IASummit2016
Shiftingpatternsofawareness
Framesofmeaning
Rocket contrail by Seay Photography via Creative Commons
He challenges us as designers. How far can we go in understanding these shifting patterns of awareness and frames of meaning and apply that to our craft?
Photo credit: Barcroft Media via Daily Mail, 3D image of human brain connections
That these beautiful brains we walk around with have somehow become disconnected from the surrounding environment, the very thing that, over hundreds of thousands of years of human evolution, tuned us as tribal hunter-gatherer poets, is…a little odd.
Photo credit: Pasos Adalente
That we somehow take stuff from our senses, process that so we can recreate it inside our brains, process it some more so we can manipulate it into calculated decisions which are then output as behaviors, is a very unnatural way to think about meaning.
From The world is the screen: elements of information environments, Andrew Hinton, 2013 IA Summit
I’m ever grateful to Andrew Hinton for showing this slide in a talk he gave in 2013 giving us another way to think about meaning, from our midcentury JJ Gibson, that puts us and our brains back on the ground, immersed in our environment. Meaning doesn’t happen as an isolated thing only inside our brains; we’re soaking in it.
Meaningisecological
J.J.Gibson1979
From our midcentury J.J. Gibson, we see that meaning is ecological because information is ecological.
Visual Informationisecological
Chair image credit: Yoichi Yamamoto
surfacesedges
textures
We don’t have to have seen a chair from every possible perspective in every possible light to recognize the chair. It’s the relationships among the surfaces, edges, and textures that we recognize, that we pick up as invariant, that let us recognize the chair.
All Informationisecological
Perceptualinformation*
Language
words ona surface
words through the air
SabrinaGolanka
All perceptual information is ecological. And present-day embodied cognitive psychologists, in particular, Sabrina Golanka, are drawing on Gibson’s ecological psychology to frame how language, or conceptual information, though different from perceptual information, is ecological too. See Sabrina Golonka (2015) Laws and Conventions in Language-Related Behaviors, Ecological Psychology, 27:3, 236-250, DOI: 10.1080/10407413.2015.1068654
Meaningemerges
InformationGoal-DirectedActions
Meaning
Meaning emerges when we act with the information in the world around us.
Meaningisdynamic
InformationGoal-DirectedActions
But this is not a static thing. Our goals change, our actions change, and the information in the environment changes. These things must adapt together. Meaning is dynamic, meaning is an event that unfolds over time.
MeaningModes
As designers, if we make deliberate decisions about these properties, then we can facilitate particular meaning modes to suit our design needs.
Impact of a drop of water on a water surface by Roger McLassus https://commons.wikimedia.org/wiki/File:2006-01-28_Drop- impact.jpg
Isinformationlikewater?
To get at our first property of meaning modes, we ask the question: is information like water?
temperature
pressure
solid
gas
liquid
Thephase-spaceofwater
If we run all the permutations on the different relative amounts of temperature and pressure, we end up with what we know as the phase-space of water. The nature of water is drastically different as a solid vs. liquid or gas.
Perceptualinformation
Language
Thephase-spaceofinformation
words ona surface
words through the air
*What if we do the same thing for the two types of information: perceptual and linguistic? What is it like to interact with different combinations of perceptual and linguistic information? Do we find neighborhoods of combinations in which it is drastically different to engage with the meaning of information?
Perceptionflowseasily Tacit
Reflexive
photo credit: Jason Pratt via Creative Commons
Perceptual information flows easily. It is tacit and reflexive. Once we’ve learned to detect it, we don’t have think about it to engage with it. Perception flows easily. It has a low viscosity, like water.
Languageisviscous,moreworktoflow Awareness
Associativity
Requiresattention
photo credit: edibleoffice via Creative Commons
Language is viscous and takes more work to flow. It is laden with awareness and associativity, and requires active attention for us to engage it.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
Easeofflowofmeaning
Generally, the area dominated by perceptual information is reflexive. The area dominated by linguistic information is more attentive. But we can get more granular than that. If we have little or no language, all perceptual information, we are engaging with meaning in a very visceral manner. If we look at the area with little perceptual information, we are operating in the world of concepts. We are thinking conceptually to engage meaning. If we have a lot of language, especially if it’s abstract, we need intense concentration in this highest viscosity state. If we have a lot of perceptual information to deal with, we must engage with intense coordination. And when we are faced with a lot of information of either type, we may become overwhelmed.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
Beforeimages Afterimages
Remember Twitter before images? This is a waybackmachine snapshot from 2011 [left]. Tweets were made of words and, because we are context jumping from one unrelated tweet to the next, it is a highly viscous way to engage with meaning. When Twitter introduced images to tweets, that disrupted the behaviors we had honed to engage with the meaning of twitter. Suddenly there were these perceptual swaths mixed in with the text. We got used to it, we developed new behavior for engaging Twitter, and we probably don’t think about it at all anymore, but Twitter changed. It phase-shifted. The nature of engaging with meaning on Twitter is not the same.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
Palette Gear
The speaker yesterday showed us Palette Gear, physical controls that can be mapped to actions on tools like Light Room. This design phase-shifts what it’s like to engage with meaningful action in Light Room: it adds more perceptual information. It requires more coordination to engage, but lets photographers engage multiple simultaneous actions and do adjust the qualities of photos faster.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
VISCOSITY TOLERANCE
PERSISTENCE PRESENCE
Behaviorprecision
That’s our first property of meaning modes: viscosity, or the ease of flow of meaning. Next, we’ll look at tolerance, or how precise our behavior must be to engage the information in a design to maintain the flow of meaning.
WIDEToleranceForgiving
Information
NARROWTolerance
Precise
Breaks the flow of meaning
Information
Toleranceisaboutprecision
A design has wide tolerance if we can veer our behavior a bit and still maintain our engagement with the information. It’s forgiving. If a design has narrow tolerance, we must be very precise with our behavior and if we veer too much, we break the engagement with the information and break the flow of meaning.
You know what to do; you reach in the direction of the thing, tapping around on the glassy surface, trying to make the sound stop. The sound is still going, so you have to you to lift your head a little, open your eyes a little to see onto the bedside table and finally manage to hit the spot.
That snooze button on your phone may be plenty big to tap with your finger when you’re awake and have your vision light adjusted and are looking at the face of the phone at a direct angle. But, the intent of this snooze button, the meaningful engagement with this snooze button, is to extend the continuity of your dream state 8 minutes and 59 seconds longer. Struggling to find a button in the dark, lifting your head a bit, opening your eyes some to try to see, breaks the flow of meaning of extending your dream state. Or worse, in your tapping around, you may accidentally hit stop and break capped nature of the flow of meaning of snoozing.
NARROWTolerancePrecise
Imprecisionasadesignvalue
WIDE ToleranceForgiving
Well-lit,Wide-awakebehavior
Night-blind,Dreamstatebehavior
The point of snooze is to keep us in our night-blind, dream state. But that requires a design that is forgiving of the kind of behavior we are capable of in that state: imprecise.
car UI concept and image from: http://matthaeuskrenn.com/new-car-ui/
Let’s look at a different example where imprecision can be a design value. The concept for a car dashboard control panel is just a blank screen. The human puts a finger anywhere on the screen and the UI comes to that spot. Depending on the number of fingers on the screen, the human can control different things: temperature, volume, etc. The human moves her finger generally up or down to make adjustments to the ambient comfort of the car.
Imprecisionasadesignvalue
NARROWTolerancePrecise
WIDE ToleranceForgiving
This design has a wide tolerance and is very forgiving of the distracted nature of a driver of a car.
Let’s look at another example of a the common button layout design. You may have some perceptual memory that the climate control button is in the lower right…
But you still have to glance in order to see exactly where you need to press to hit the button and get to the climate controls.
But you still must glance to bring your finger to the exact button location. And, if you want to adjust quite a bit, you must lift your finger off the button and return it to the exact spot to press several degrees, at the same skewed angle with respect to where you are sitting and facing in the driver’s seat.
Viscosity&Toleranceworktogether
Tolerance
Viscosity
Tolerance
Viscosity
The viscosity is quite low for both of these designs, but the difference in tolerance makes or breaks the flow of meaning for driving a car.
Flowofmeaningofdrivingacar
Swathofattention
Peripheryofdistraction
Wayfinding
PathEvents
Music/News
Passengers
The flow of meaning of driving a car involves wayfinding, continuously watching our path; watching out for events in our path that we must react to; we may be listening to music or news, and that can waft in and out of our attention; and we may be chatting with out passengers.
Flowofmeaningofdrivingacar
Swathofattention
Peripheryofdistraction
Wayfinding
PathEvents
Music/News
Passengers
Comfort
The need to glance at the buttons to engage their precision takes our vision off of watching for wayfinding cues and path events and breaks the flow of meaning of driving a car.
Flowofmeaningofdrivingacar
Swathofattention
Peripheryofdistraction
Wayfinding
PathEvents
Music/News
PassengersComfort
Voicecommands
What about forgetting the screen altogether and using voice commands? That uses different type of information, entirely different phase-space neighborhood (conceptual instead of visceral) and allows us to focus on the visual information we need for wayfinding and reacting to path events. But, it’s a higher viscosity and requires our active attention, higher still because we have to remember the precise words and speech syntax to verbally engage the environment controls of the system. So, we have to break our attention to listening to news/music and chatting with our passengers. We’re still breaking the flow of meaning of driving a car. We’re breaking less critical parts of that flow, but still.
Flowofmeaningofdrivingacar
Swathofattention
Peripheryofdistraction
Wayfinding
PathEvents
Music/News
Passengers
Comfort
The concept control panel precision and low viscosity allows us to engage it in the periphery of our distraction, and maintain the flow of meaning of driving a car.
Twosidestotolerance
HUMANBEHAVIOR
SYSTEMBEHAVIOR
WIDE NARROW
TEMP VOL FAN
But, there are two sides to tolerance. We’ve looked at the precision required for the human’s behavior. The design also has behavior and that has a tolerance too. We see that human engagement has a tolerance and system response has a tolerance. It’s not one or the other alone that accounts for the the tolerance of the meaning mode, but both feeding off of each other. Even though the human can veer a bit in how she engages the car control panel, the system doesn’t make wild changes in temp or volume. Temp still goes up and down by degree, and volume changes by a fine increment, and the fan speeds shift specifically. So we see that, the system has a precise behavior even though the human is allowed to have imprecise behavior to engage with it.
Twosidestotolerance
HUMANBEHAVIOR
SYSTEMBEHAVIOR
WIDE-NARROW
Forgiving-Precise
TEMP VOL FAN
We’d call the overall human-system behavioral tolerance a forgiving-precise dynamic.
HUMANBEHAVIOR—SYSTEMBEHAVIOR
WIDE-NARROW
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW NARROW-WIDE
WIDE-WIDE
Comfort
The forgiving-precise dynamic is just one of the combinations.
HUMANBEHAVIOR—SYSTEMBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
CODING
NARROW-WIDE
WIDE-WIDE
An example of a precise-precise, human-system dynamic is writing computer code. If we think about the flow of meaning of writing code, it’s a very high viscosity activity requiring lots of intense concentration to wield all of the abstractions; encode them in the precise syntax conventions of the particular language
HUMANBEHAVIOR—DESIGNBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
CODING
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
Long-lead discovery
Play/explore multiple possible worlds
Code Review image from GitHub
It’s mostly made of concepts and very persnickety syntax. There is some perceptual information in the nested commands, and some color coding for code reviews in some cases. The human must be extraordinarily precise with this language; the system behaves exactly as dictated in the code.
HUMANBEHAVIOR—DESIGNBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
3DMODELING
Mouse image credit: 3D Connecxion Model Image credit: MaTane Tutorials, Autodesk Inventor Mirroring, via YouTube
Where I work at Autodesk, we make tools that help engineers and designers create things in the world, products, machinery, architecture, by way of 3D modeling.
HUMANBEHAVIOR—DESIGNBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
Long-lead discovery
Play/explore multiple possible worlds
3DMODELING
Mouse image credit: 3D Connecxion Model Image credit: MaTane Tutorials, Autodesk Inventor Mirroring, via YouTube
Direct 3D modeling requires high precision behavior to select particular edges and faces and points on the geometry of the thing being made, and then make very precise adjustments to them. The system updates the visualization very precisely with each action. A lot of our customers use an exquisitely precise 3D mouse. This requires a lot of perceptual coordination and is a very visceral way of engaging with information.
HUMANBEHAVIOR—DESIGNBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
Long-lead discovery
Play/explore multiple possible worlds
3DMODELING
Mouse image credit: 3D Connecxion
DigitalAgent
What about when we start offloading some of the modeling work to digital agents? The word agent implies agency, or the ability to act with some degree of freedom.
HUMANBEHAVIOR—DESIGNBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
Long-lead discovery
Play/explore multiple possible worlds
DigitalAgent
Constraints
Instead of the human doing all the intricate geometry sculpting, she sets some constraints on what we’re making, and then the digital agent runs through all the different possibilities within those constraints and brings them back to the human to refine further.
Generative design image credit: Autodesk project Dreamcatcher
The digital agent can then visualize all the possibilities within the human’s set constraints, and the human can make further adjustments on constraints. If the designer wants to explore options for chair design, the constraints may be a particular type of material, amount of material, requirements for weight that should be supported, that it should still have 4 legs. And the the agent comes back with the range of possibilities that the designer can filter further.
Generative design image credit: Autodesk Airbus partition project
Or in designing a partition for a jet plane that is much lighter, uses less material, but still meets very important stress criteria.
HUMANBEHAVIOR—SYSTEMBEHAVIOR
NARROW-WIDE
Precise-Fuzzy
Quick-hit discovery
Unknown possibilities
GENERATIVEDESIGN
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
3DMODELING
This type of generative design phase shifts our customers from the highly perceptual engagement with the direct geometry of the design that requires intense coordination in a precise-precise style of engagement between human and design tool, over to a conceptual mode of engaging in which our customers are no longer directly conduct the actual geometry of the design itself. Instead they precisely constrain the conceptual facets of the design and offload the geometry decisions to the design tool as digital agent. The tool becomes an agent that has agency to run through permutations and come up with possibilities for the designer to then refine further. These two different styles of modeling have different meaning modes in terms of viscosity and tolerance.
HUMANBEHAVIOR—SYSTEMBEHAVIOR
NARROW-WIDE
Precise-Fuzzy
Quick-hit discovery
Unknown possibilities
GENERATIVEDESIGN
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
Long-lead discovery
Play/explore multiple possible worlds
3DMODELING
MACHINELEARNING
If we add some machine learning to the digital agent’s agency, then we open up the wide-wide tolerance, or forgiving-fuzzy dynamic for 3D modeling. The agent can tolerate less precision on our part because it is learning from other things like usage patterns, material properties, all sorts of things that can expand into feedback loops leaving the human to task of making tweaks as desired based on aesthetics or higher-level requirements.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
NARROW-WIDE
Precise-Fuzzy
Forgiving-Precise
NARROW-NARROW
Precise-Precise
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
VISCOSITY TOLERANCE
PERSISTENCE PRESENCE
Coordinatingvs.Adjusting
Our next property is persistence: are we relying on the information in the design to help coordinate our behavior, or are we using it to make adjustments or decisions? (This property is based on: Sabrina Golonka (2015) Laws and Conventions in Language-Related Behaviors, Ecological Psychology, 27:3, 236-250, DOI: 10.1080/10407413.2015.1068654 )
COORDINATINGBEHAVIOR ADJUSTING
HUMANBEHAVIOR—SYSTEMBEHAVIOR
Forgiving-Precise
Adjusting while distracted
Multi-tasking
NARROW-NARROW
Precise-Precise
Fine-tuning
Rangeof complex actions
NARROW-WIDE
WIDE-NARROW
3DMODELING Precise-Fuzzy
Quick-hit discovery
Unknown possibilities
GENERATIVEDESIGN
WIDE-WIDE
Forgiving-Fuzzy
Long-lead discovery
Play/explore multiple possible worlds
Notice that in 3D modeling, the human is using the design tool to coordinate behavior over time as she directly sculpts the geometry. In generative design, the human is simply making adjustments, decisions about which of the possibilities best suit the situation and constraints. This difference between relying on information to coordinate our behavior vs. to make adjustments is our next property of meaning modes. (Notice that when precision is about conceptual information (or the language-dominated region of the phase-space), precision is not about usability of the interface itself; the literal act of engaging, but in the conceptual act of engaging. How conceptually precise must the human be to engage the information and make adjustments (in the case of generative design). In the case of controlling the car dashboard panel, the questions of tolerance (precision) involved how precise the human had to be in physical behavior to engage the information. This is an important distinction.)
COORDINATINGBEHAVIOR ADJUSTING
Three Dancers, pen and ink by Chris Carter
When we think about gestural input, which one is that suited to? Which style of persistence?
ADJUSTING
Hololens gesture image credit: Microsoft
GestureasCONCEPT
Forgiving-Precise
When I make gesture with my arm or hand or eye gaze, I’m not able to do it exactly the same angle and speed, etc., every time; there’s too many degrees of freedom. There has to be some invariant structure the system picks up to get the gesture. That makes them a lot like concepts. Directions; adjustments.
COORDINATINGBEHAVIOR
Sonar haptics image credit: MIT Media Lab
GestureasSCULPTING
Precise-Precise
MIT and other groups are starting to use focused sound ways to create a mechanical, haptic effect that we can feel. We could actually touch and feel holograms. This is just proof of concept now, but we could imagine that it could then use our behavior, our gestures, to directly change it. We could actually use gestures to coordinate our behavior along with the haptic feedback we get from the focused sound waves changing as our motions change the hologram. That shifts what we can do with gestures from just adjusting by evoking concepts, to relying on the system to coordinate our behavior over time.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
NARROW-WIDE
Precise-Fuzzy
Forgiving-Precise
NARROW-NARROW
Precise-Precise
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
COORDINATINGBEHAVIOR ADJUSTING
VISCOSITY TOLERANCE
PERSISTENCE PRESENCE
SenseofPlace
Persistence is about whether we are relying on the information in the system to help coordinate our behavior over time, or to make adjustments. Our next property is Presence or sense of place.
Creative Commonsfreeusageforestpanoramaimage
From the ecological point of view, place is a collection of affordances that support a meaningful activity (NOTE: see Heft, The Participatory Character of Landscape https://www.researchgate.net/publication/253950787_The_participatory_character_of_landscape ). This forest can be many different places, depending on the activity. If I’m looking to set up camp, place becomes a particular layout of the forest that offers a clearing, a reasonably unbumpy ground, and perhaps a break in the trees to watch the stars.
Creative Commonsfreeusageforestpanoramaimage
When I’m ready to make a camp fire, place becomes an extent of the forest floor within a walkable radius from my camp. The affordances of this place become objects that appear twig-ish and stick-ish, with surfaces that appear dry and burnable. A forest contains nested places made of physical information, depending on my activity. We also have places made of digital information. For more, see Information Architecture for the Web and Beyond, Morville, Rosenfeld, Arango, and Pervasive Information Architecture, Resmini and Rosati, and Understanding Context, Andrew Hinton.
Precise-Fuzzy
NARROW-WIDE
ADJUSTING
VISCOSITY TOLERANCE PERSISTENCE
One place made of digital information is Amazon. The Amazon website has a viscosity that is a combination of concepts in terms of the categories of products and their facets, and the perceptual cues of the products themselves displayed as images. If you are browsing boardgames or books, you may lean more on the labels, titles, and recommendation text for the flow of meaning; if you are looking for a new shirt, you may engage more heavily on the perceptual information about the surfaces, edges, textures of the shirts portrayed in the images. Amazon has a precise-fuzzy human-system dynamic in that the human must select categories or enter well-formed search terms, but the system is fuzzy in offering “customers also bought” and customer reviews and rec’s for other things, and search term structuring with each additional character typed. The human relys on the information in Amazon to adjust, and make decisions about things to purchase.
MARKETPLACE
If we zoom out, this set of activities we engage in Amazon gives it a sense of place. We recognize Amazon as a marketplace.
DIGITALPHYSICAL
MARKETPLACE
Emergence
Imagecredit:RDRiccoboni
We know there is some emergent nature to engaging a digital place like Amazon in whatever our present physical surroundings may be, and the designers of Amazon have to worry about its meaning shape shifting across contexts of devices, and languages, and accessibility modes, but the physicality of the surrounding environment, the objects and layout of the surrounding physical environment, do not participate directly in the meaning of engaging amazon as marketplace. As far as meaningful engagement with the activities in the marketplace that is amazon, our physical surroundings may as well be a painting.
DIGITALPHYSICAL
There is a designed connection between the physical box that contains the thing you bought, and the digital object that is order tracking of the thing you bought in the marketplace. Tracking your box location is part of the meaningful activities in Amazon as Marketplace, and participates in the place that is amazon.
DIGITALPHYSICAL
Correspondence
--DavidBenyon,BlendedSpaces
PRESENCE
These designed connections among physical and digital objects and relationships are what David Benyon, working on Blended Spaces in HCI, calls correspondences. This extends the presence of Amazon marketplace to our physical world. Although the meaningful activities we engage in this marketplace are mostly digital, this connection extends the meaningful activities, our sense of presence, our sense of place of Amazon, into the physical world. These designed connections among physical and digital objects and relationships are what David Benyon calls correspondences. See his research including, http://iwc.oxfordjournals.org/content/24/4/219.short (Also, keep an eye on the work of Andrea Resmini, bringing these concepts to his Pervasive Information Architecture work.)
DIGITALPHYSICAL
Roomimagecredit:AutodeskRevitrendering;VRgamecredit:SonyProjectMorpheus
VR
What about virtual reality?
DIGITALPHYSICAL
VR
Not only does virtual reality deliberately not offer correspondences to our physical surroundings, but deliberately tries to obliterate our ability to shift our attention from the digital place to our physical surroundings.
DIGITALPHYSICAL
Correspondence
Wallalert
VR
But, we like to move around. And because we can’t perceive our physical surroundings, we might run into walls or furniture. So, we have to let a little physicality seep into our “virtual” reality. These “wall alerts” take the form of beeping sounds or maybe a visual cue that appears like an aurora in this other place that we are inhabiting. This wall alert is a designed correspondence too, but rather than participating directly in the flow of meaning in the digital place, it disrupts that presence and for a moment asks our presence to co-exist in two parallel places. It’s a very unnatural kind of presence.
DIGITALPHYSICAL
Holograms
BALANCE
LOCOMOTION
OBJECTS
NoCorrespondences
LAYOUT
Now we can see through to the physical surroundings. Balance and locomotion, and visual information about the layout of physical surroundings is available, but the layout and objects in the physical surroundings don’t participate in the meaning of engaging with the hologram. There’s no correspondence there. Although we can shift attention between the digital place that is the hologram and its meaningful activities, and my physical surroundings, they are not both participating in the same sense of presence.
Imagecredit:Microsoft,Hololens gameFragments
In the game Fragments for Hololens, the system first 3D scans your entire room. The objects in the room actually participate in the game. A detective game where you find clues. So a hologram character can sit on your actual chair, because the chair is recognized, and has affordances for the holograms. Theoretically, the system could place digital things inside physical drawers and cabinets because they could be recognized in terms of the dispositions of the kinds of activities they support.
DIGITALPHYSICAL
3DRoomScanningMixedReality
Correspondence
OBJECTS
LAYOUT
PRESENCEspansboth
In the case of 3D room scanning mixed reality (or whatever fancy name gets assigned to that), the physical objects and layout of the room participate directly in the meaning of the place, and there is a designed correspondence between the physical place and digital place, and our sense of presence spans both.
DIGITALPHYSICAL
DegreeofPhysical-Digital
Correspondences
PRESENCE
Nouns&VerbsAffirmationsextendingEcosystem
-AlaineMackenzie -DanBrown
OBJECTS+INTENT
So, one important consideration for presence is recognizing the degree to which our physical surroundings participate in the meaningful activities of the designed place. Are our physical surroundings (objects and layout) participating in the place we’re engaging, in the meaningful activities we’re engaging, or are they just our surroundings? Two of the speakers yesterday, Alaine Mackenzie’s “nouns and verbs” and Dan Brown’s affirmations extending the ecosystem of the domain, are great ways to help us define what are our meaningful correspondences among physical objects/layout and digital objects/layout? Or, if they need not be present. Defining the ontology of our design domain is a critical step in understanding the scope of place (and therefore of presence when our humans engage them). (For more on ontology, Dan Klyn on ontology-taxonomy-choreography, and Abby Covert’s book, How to Make Sense of Any Mess.)
“Oneseestheenvironmentnotjustwiththeeyesbutwiththeeyesintheheadontheshouldersofabodythatgetsabout”
Imagecredit:Designboom (AlisonBrooksArchitects‘TheSmile’)
J.J.Gibson
We humans want to explore! We don’t just look at our surroundings, we explore them to meaningfully engage them. From an ecological point of view, we see the environment by moving through it. J.J. Gibson quote from, An Ecological Approach to Visual Perception, p.222 1979. According to JJ Gibson, our legs (or whatever we use to physically move around) are part of our visual systems. It’s part of how we see. See also Heft (2010) “Affordances and the perception of landscape: an inquiry into environmental perception and aesthetics,” in Innovative Approaches to Researching Landscape and Health. An ecological point of view makes a strong distinction between looking at an image and engaging the environment.
DIGITALPHYSICAL
DegreeofPhysical-Digital
Correspondences
PRESENCE
ExplorabilityofObjects&Layout
ACTINGWITH
LOOKINGAT
We can ask in our designs, to what degree are there correspondences between the objects and events in our digital place and physical surroundings? How explorable are the objects in our designs? If your augmented reality design is just an informational overlay on an object in a specific geographic location, that’s just a map; a distributed map, but still just a map; it doesn’t have materiality because it isn’t explorable.
WRITINGGROUP
MARKETPLACE
Explorability is not just for physical, “virtual” places. Around 2006, new kinds of digital places popped up, nested within the digital place that is Amazon the marketplace. Some shoppers found humor in some of the products Amazon offers, like Tuscan whole milk and microwave for one. These shoppers re-purposed the product reviews to write humorous literary works, short stories, poems, about the products. Others would riff on these works and add their own creative writing. Shoppers became writers. The product reviews of these products became writing groups, and nested places with distinct meaningful activities very different from Amazon as marketplace, yet within.
*'t?'t?'t?'t? Paradise Redredged
By Dr. D. v. Sim mental on August 11, 2006
Timeless works often suffer at the hands of translators. One thinks of the numerous and continuing attempts to render Dante's "Divina Commedia" (another early vernacular Italian masterpiece and contemporary to the justifiably obscure Tuscana Latte series) and the struggle with both terza rima and meaning. No so for Tuscan Whole M ilk ("La tutta
latte"). Few works are better left translation of the redredged thir librettist for two of the earliest p Tuscan series and "Mechanicae
** * * * Milk for a Better Tomorrow
By C.A. Little on August 29, 2006
opening nights sometime in the I feel compelled to relay my story with Tuscan Whole Milk. My husband is a middle manager store. In an attempt to move him up the ranks of paper pushers (pardon the pun), we decide
That Stilton translated the Tus and invite some of his bosses. Not being much of a homemaker, I was frantic to come up wi1 Emmenthaler, present at the Gr I discovered the response people seem to have to Tuscan Whole Milk. Reisen", returning to his rooms I the Arno , but had been clogged . I ordered a gallon - 128 fluid ounces of pure dairy goodness. I was quite concerned about he
massive s
complete
work.He
and he fle
� One Friday, Without the Milk, October 30, 2006 By Amazon Customer This review Is from: Tuscan Dairy Whole Vitamin D Milk, Gallon, 128 oz (Grocery)
He always brought home milk on Friday.
After a long hard week full of days he would burst through the door, his fatigue hidden behind a smile. There was an icy jug of Tuscan Whole Milk, 1 Gallon, 128 fl oz in his right hand. With his left hand he would grip my waist - I was always cooking dinner - and press the cold frostiness of the jug against my arm as he kissed my cheek. I would jump, mostly to gratify him after a time, and smile lovingly at him. He was a good man, a wonderful husband who always brought the milk on Friday, Tuscan Whole Milk, 1 Gallon, 128 fl oz.
ed to se, using Tu
greet o the kitct
edel Crys I wrought Milk. lwa, ilk! I COUit
sticated an,
*'Clf:rtrC:i With apologies to Emily
By S. Mayo on February 1, 2011
Because I could not stop for milk, It kindly came to me. I ordered it through Amazon, Whole milk; sweet Tuscany.
The driver, though, he knew no haste, No Expediter, he! The wait: Four days, and weekend, too. It seemed eternity.
My milk, it passed through NYC Where traffic snarled in rings, Slow past the fields of grazing cows Who nodded knowingly.
Its expiration date passed, too; And though the truck was chill, Vile germs digested lactose sweet And had their gruesome fill.
Rarely mi read (if ne
diatribes
infamous
Grasso") by some s
Dickinson' Proust's " Neal; "El Peck and
Then there was that Friday, the terrible Friday that would ruin every Friday for the rest of my life. The It paused before my house at last, door opened, but there was no bouyant greeting - no cold jug against the back of my arm. There was can Whole Milk. 1 had to place my
This Stilto
the flavor spoke littl is not cert
One start
• 3comm
no Tuscan Whole Milk in his right hand, nor his left. There came no kiss. I watched as he sat down in (By USPS Ground) a kitchen chair to remove his shoes. He wore no fatigue, but also no smile. I didn't spe'!fo��e!!:!:�=:::!::s:::o:::f:::th=e=fi=m=e=d=a=i�s=a=v=a=n=ts=a=t=T=u=s=ca=n=::::!:=======�===================i back to the beans I had been stirring. I stirred until most of their little shrivelled skins surface of the cloudy water. Something was wrong, but it was vague wrongness that no ar hard thought could give shape to. • Over dinner that night I casually inserted,"What happened to the milk?"
"Oh,"he smiled sheepishly, glancing aside,"! guess I forgot today."
That was when I knew. He was tired of this life with me, tired of bringing home the Tus r Milk, 1 Gallon, 128 fl oz. He was probably shoveling funds into a secret bank account, I I
228
votes
Question:
Answer:
If I spill it, can I cry?
If you do it is best to cry either next to it or below it. Crying over it is useless . By Marcus M . on February 25, 2013
� See more answers (7)
apartments in town, casting furtive glances at cashiers and secretaries and waitresses. l,'h,'-��---=--=��=�-----��------�--------------------'
knew it was over. Some time later he moved in with a cashier from the Food Mart down the street. Tuscan Whole Milk, tomorrow a
And me? Well, I've gone soy. Help other customers find the most helpful reviews
Was this review helpful to you? GisJ � Yes � Report abuse
-S&J
• 7 comments 602 people found this helpful. J
The Tuscan whole milk writing group has its own activities embedded within the product reviews. The act of writing a review, the act of selecting stars, rating a review as ‘helpful’ are all activities with very particular meaning in this nested place. It has its own meaning mode that is very distinct from the meaning mode of Amazon as marketplace.
MARKETPLACETUSCANMILK
WRITINGGROUP
ADJUSTINGWIDE-WIDE
Forgiving-Fuzzy
ACTINGWITHDIGITALPHYSICAL
NoCorrespondences Explorable
The Tuscan milk writing group has its own meaning mode. It has a high viscosity: highly conceptual, made of language, creative writing, playing with literary genres and tropes, and riffing off of the works of others with the dual purpose of literary form and humor. We’d call this a forgiving-fuzzy human-system dynamic because a review writer participating in this writing group can introduce a new trope, a new way of riffing on others that other writers can then riff off of in turn. The forgiving-fuzzy dynamic has vast generative power to associate on the work of others and weave in new literary styles. Ten years later, people are still writing Tuscan milk reviews. Engaging the information in this place involves writers relying on the information to adjust their behavior, responding creatively to what is already there, what just appeared, how other writers comment on their work. There are no designed correspondences between the digital place and physical surroundings: the meaning, the set of activities that is this place all occur in the digital layer. This place is highly explorable: writers in this place are acting with the information they engage.
SELECTSTARS
COMMENTONOTHERS
MARKREVIEWS‘HELPFUL’
ANSWERQUESTIONS
WRITEREVIEWSASSHORTSTORIES,PARABLES,POEMS
Writers engage in a variety of meaningful activities that make this an explorable place.
MARKETPLACETUSCANMILK
WRITINGGROUP
ADJUSTINGWIDE-WIDE
Forgiving-Fuzzy
LOOKINGATDIGITALPHYSICAL
NoCorrespondencesPhotostatic
NOTSIGNEDIN
Notice that the explorability changes if the writer is not signed in. Only signed in writers can comment on the writings of others, mark reviews ’helpful,’ and write their own reviews. When a writer visits this place but is not signed in, the place loses its materiality; the writer can only look at this place, she cannot engage the meaningful activities. It’s photostatic, like a painting. This changes the nature of how a writer finds presence in this place.
intenseconcentration
intensecoordination
concep
tual
visceral
Perceptualinformation
Language
NARROW-WIDE
Precise-Fuzzy
Forgiving-Precise
NARROW-NARROW
Precise-Precise
WIDE-NARROW
NARROW-WIDE
WIDE-WIDE
Forgiving-Fuzzy
COORDINATINGBEHAVIOR ADJUSTING
Physical-Digital
Correspondences ACTINGWITH
LOOKINGAT ExplorabilityofObjects&Layout
VISCOSITY TOLERANCE
PERSISTENCE PRESENCEDIGITALPHYSICAL
These are four properties of the invisible things about meaning we can recognize in our designs. We can use viscosity, or ease of the flow of meaning, like a design dial to speed up an interaction or deliberately slow it down and draw awareness and attention when something important needs it, or something engaging that we don’t want to miss. Tolerance is about considering the precision of the human-system behavior dynamic, and which situations those different dynamics are good for. Persistence is about whether we are relying on the information to help coordinate our behavior as we act over time, or if we are using it to make adjustments or decisions. Presence is about our sense of place, and recognizing when we need designed correspondences among our physical surroundings and our digital places and what objects and layouts are participating in meaningful ways in our place. Presence is also about how explorable the objects and layout of digital and physical surroundings are in allowing us to act with them, or just look at them (photostatic). All of these together facilitate very different meaning modes for the humans that engage them.
MeaningModes
Rocket contrail by Seay Photography via Creative Commons
These properties of meaning are really abstract things. Why should designers add this to all the other things we already have to worry about in our designs?
J.J.Gibson’sObserverisnotin1979anymore
We’ve opened many more wormholes to information with our designs. [Watch icon via bluetip design; VR headset via Michael Beno; iphone icon via Edward Boatman; shoe icon via romzicon; egg carton via Blaise Sewell; toaster icon via Creative Stall; band icon via Nate Eul]
Informationwormholesaregettingalloverus
And they’re getting all over us; in our clothes, on our kids, and all over the objects around us.
Robotsmindingallthewormholes
Samedegreeofattention
Inthesameway
We don’t want to have to act like robots minding all the information wormholes with the same degree of attention, in the same way.