![Page 1: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/1.jpg)
Direct Manipulationand Instrumental Interaction
CS349 -- Direct Manipulation1
![Page 2: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/2.jpg)
Rev
iew
: Int
erac
tion
vs. I
nter
face • What’s the difference between user interaction
and user interface?
• Interface refers to what the system presents to the user– it’s what you can manipulate and what the system uses to
present feedback
• Interaction refers to the sequence of actions a person expresses and the corresponding system responses– it unfolds over time
CS349 -- Direct Manipulation2
![Page 3: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/3.jpg)
Rec
all:
Inte
ract
ion
Para
digm
s • Batch interfaces– Knobs and Switches– Punch Cards
• Conversational interfaces– Command Line– Speech– Gestures
• Graphical interfaces– Direct Manipulation– …
CS349 -- Direct Manipulation3
![Page 4: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/4.jpg)
Rec
all:
Gra
phic
al U
ser I
nter
face • Hardware interface
– High resolution, high refresh graphics display– Keyboard– Pointing device (e.g., mouse)
• WIMP interface– Windows, Icons, Menus,
and Pointer– Desktop, files and folders
4 CS349 -- History
![Page 5: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/5.jpg)
Wha
t is
Dire
ct M
anip
ulat
ion? Graphical User Interfaces often use metaphor to suggest how a system
should be used.• e.g. file folders on a desktop suggest specific actions: you can open
a folder to reveal files, which you can read.
Affordance: • a quality of an object or of an environment which allows one to
perform an action• Don Norman: an affordance should suggest its uses; uses
should be “discoverable”
Direct Manipulation is an attempt to make build affordances into the interface that resemble affordances for real world objects.
CS349 -- Direct Manipulation5
![Page 6: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/6.jpg)
Wha
t is
Dire
ct M
anip
ulat
ion? By Example:
– Dragging a document to the trash, or to the printer – Opening a file folder to reveal files that can be read– Changing the size of a shape by dragging a “handle”– Inserting characters in a document by pointing to where
they should go (with a mouse/cursor/insertion point) and then typing
– “Dialing” a phone number using a virtual keypad– Playing a song using controls like a physical CD/DVD player
(which were like tape decks, which were like some reel-to-reel tape recorders, …)
CS349 -- Direct Manipulation6
![Page 7: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/7.jpg)
Wha
t is
Dire
ct M
anip
ulat
ion? By Contrast: It’s one of several interaction styles
– Menu selection– Fill-in forms– Command language– Natural language– Direct manipulation– …
• Used on desktop systems, touch-based and mobile systems.
CS349 -- Direct Manipulation7
![Page 8: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/8.jpg)
Wha
t is
Dire
ct M
anip
ulat
ion? By Characteristics:
– There is a visible and continuous representation of the task objects and their actions. Consequently, there is little syntax to remember.
– The task objects are manipulated by physical actions, such as clicking or dragging, rather than by entering complex syntax.
– Every operation is syntactically legal– Operations are rapid and incremental– The effect of operations on task objects are immediately
visible– (Almost) all actions are reversible– Users can explore without severe consequences
CS349 -- Direct Manipulation8
![Page 9: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/9.jpg)
Wha
t is
Dire
ct M
anip
ulat
ion? By Benefits:
– While interacting with DM interfaces, users feel as if they are interacting with the task object rather than with the interface, so they focus on the task rather than on the technology. There is a feeling of direct involvement with a world of task objects rather than communication with an intermediary.
CS349 -- Direct Manipulation9
![Page 10: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/10.jpg)
Bret Victor, Inventing on Principle (talk from CUSEC 2012)– http://vimeo.com/36579366 2:33 – 4:30 (2:30 – 5:25 for animation)
CS349 -- Direct Manipulation10
![Page 11: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/11.jpg)
Evaluating Graphical InterfacesInstrumental Interaction
CS349 -- Direct Manipulation11
![Page 12: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/12.jpg)
Look
and
Fee
l Look = How manipulatable objects are presented in the interface
Feel = How user expression of these objects is translated into commands to manipulate underlying data
12
express
presentperceive
translate(input)
(output)
CS349 -- Touch Interfaces12
![Page 13: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/13.jpg)
Inte
ract
ion
Mod
el “An interaction model is a set of principles, rules, and properties that guide the design of an interface. It describes how to combine interaction techniques in a meaningful and consistent way and defines the look and feel of the interaction from the user's perspective. Properties of the interaction model can be used to evaluate specific interaction designs.” (Lafon, 2000)
• We have talked about execution-evaluation cycle and interaction framework in this course, but these are at the user-action level, not at the level of describing UI objects
Beaudouin-Lafon, M. 2000. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. Proceedings of CHI '00, 446-453. http://doi.acm.org/10.1145/332040.332473
CS349 -- Touch Interfaces13
![Page 14: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/14.jpg)
Inst
rum
enta
l Int
erac
tion “A model of interaction based on how we naturally use tools (or
instruments) to manipulate objects of interest in the physical world.”
An interface is composed of–Domain objects: the thing of interest,
e.g., data and associated attributes– Interaction instrument: a necessary
mediator between the user and domain objects
Domain objects are manipulated using interaction instruments.
CS349 -- Touch Interfaces14
![Page 15: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/15.jpg)
Inst
rum
ents
vs.
Dom
ain
Obj
ects
presentation object
resizeinstrument
drawingobject
scrollinstrument
magnificationinstrument
slideobject
resizeinstrument
new slideinstrument
translateinstrument
slide selectioninstrument
slide reorderinginstrument
CS349 -- Direct Manipulation15
![Page 16: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/16.jpg)
Inst
rum
ent A
ctiv
atio
n • Activation: When instrument (logical part) is under user’s control (via physical part – mouse or keyboard)– Using the scrollbar instrument – how
did it get activated?– Using the rectangle creation instrument
– how did it get activated?• WIMP instruments activated spatially or
temporally or by a combination– Spatial: caused by moving
mouse/cursor into a control area– Temporal: caused by a former action;
remains in effect until the activation of another instrument; enters a “mode”.
CS349 -- Direct Manipulation16
![Page 17: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/17.jpg)
Inst
rum
ent A
ctiv
atio
n • WIMP instruments activated spatially or temporally– Spatial: caused by
moving mouse/cursor into a control area
– Temporal: caused by a former action; remains in effect until the activation of another instrument.
• Spatial activation requires an on-screen representation.
• Temporal activation could be via on-screen or keyboard.
spatial scroll
instrument
temporalrectangle
instrument
spatial shape selection
instrument
temporal & spatial rectangle activation
instrument
spatial resize
instrumentspatial magnification
instrument
temporal & spatial rectangle selection
instrument
CS349 -- Direct Manipulation17
![Page 18: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/18.jpg)
CS349 -- Direct Manipulation18
scaleinstrument
scrollinstrument
translateinstrument
rotateinstrument
new slideinstrument
fill selected shape
instrument
![Page 19: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/19.jpg)
CS349 -- Direct Manipulation19
![Page 20: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/20.jpg)
Act
ivat
ion
Cos
ts • Spatial and Temporal activations have different costs– Spatial: cost to move the cursor to the area of interest.– Temporal: cost to activate a widget through a sequence of
actions or steps.
• Interface designers face tradeoffs because costs can become significant when users must frequently change instruments.– Option: Add more input devices
http://palettegear.comCS349 -- Direct Manipulation20
![Page 21: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/21.jpg)
Eval
uatin
g In
stru
men
ts Degree of indirection– Spatial/ temporal offset between instrument and action on
object
Degree of integration– Ratio of degrees of freedom of instrument to degrees of
freedom of input device
Degree of compatibility– Similarity of action on control device/instrument to action on
object
CS349 -- Direct Manipulation21
![Page 22: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/22.jpg)
2D measure of spatial and temporal offsets of instrument• Spatial
– Close: drag to translate, handles on rectangle to resize– Far: scrollbar, dialog boxes
• Temporal– direct dragging response
vs. waiting until after exiting dialog
Continuum between direct manipulation and indirectmanipulation Spatial Offset
Tem
pora
l Offs
et
Indirect
Manipulation
Direct
Manipulation
Handles
Drag n' DropScrollbars
Dialog Box
Inspector
Deg
ree
of In
dire
ctio
n
(image: after lafon 2000)CS349 -- Direct Manipulation22
![Page 23: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/23.jpg)
Deg
ree
of In
tegr
atio
n The ratio between the number of degrees of freedom (DOF) of the instrument and the DOF captured by input device
1 DOF3 DOF
2 DOF 2 DOF
CS349 -- Direct Manipulation23
![Page 24: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/24.jpg)
Deg
ree
of C
ompa
tibili
ty The similarity between the physical actions on the instrument and the response of the object.
Dragging = high Scrolling = medium Dialog = low
CS349 -- Direct Manipulation24
![Page 25: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/25.jpg)
Direct Manipulation Reprise
CS349 -- Direct Manipulation25
![Page 26: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/26.jpg)
Dire
ct M
anip
ulat
ion
Prin
cipl
es • There is a visible and continuous representation of the domain objects and their actions. Consequently, there is little syntax to remember.
• The instruments are manipulated by physical actions, such as clicking or dragging, rather than by entering complex syntax.
• Operations are rapid and incremental • Their effects on domain objects are immediately visible.• Reversibility of (almost) all actions• Users can explore without severe consequences• Operations are self-revealing• Syntactic correctness – every operation is legal
(from User Interface Design & Evaluation, p. 213-214)CS349 -- Direct Manipulation26
![Page 27: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/27.jpg)
GU
I !=
Dire
ct M
anip
ulat
ion • Many commands are invoked indirectly
– Menus, dialog boxes, toolbars• Not direct manipulation• They are mediators that pull users away from objects of
interest• Many objects of interest are hidden
– Style sheets– Alignment constraints are often fleeting
• Lots of objects in the interface are not objects of interest– Toolbar pallets
(From Beaudoin-Lafon [2000])CS349 -- Direct Manipulation27
![Page 28: Direct Manipulationcs349/w17/slides… · · 2017-02-19–Pointing device (e.g., mouse) •WIMP interface ... Graphical User Interfaces often use metaphorto suggest how a system](https://reader034.vdocuments.us/reader034/viewer/2022051321/5af4dd2b7f8b9a95468e8849/html5/thumbnails/28.jpg)
Dire
ct M
anip
ulat
ion
Issu
es • Visually impaired users can’t see the graphics; no linear flow for screen readers; physically impaired may have difficulty with required movements
• Often consume significant screen space, forcing valuable information off-screen.
• Switching between keyboard & pointer is time consuming• Analogies may not be clear
– Users need to learn meaning of visual representations– Visual representations may be misleading
CS349 -- Direct Manipulation28