ISTD 2003

Interactive Systems Technical Design

Lecture #3

Hybrid computing environments

ISTD 2003

Hybrid Computing Environments

• We live in the fusion of real (physical) and virtual (computer) worlds

• Some systems add virtual properties to physical world

• Some systems add physical dimension to virtual worlds (e.g. Pick-and-Drop studied here)

• These two approaches are not contradicting nor mutually exclusive• It is important to find the right balance between

virtual and physical dimensions

ISTD 2003

Case Studies

Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer

EnvironmentsSony Computer Science Laboratories, Japan

Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing

EnvironmentsSony Computer Science Laboratories, Japan

ISTD 2003

Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer

Environments

Jun Rekimoto, Sony Computer Science Laboratories, Japan

Proc. 10th Annual Symposium on User Interface Software and Technology, Banff, Canada

ISTD 2003

Pick-and-Drop: A Direct Manipulation Technique for

Multiple Computer Environments

Jun Rekimoto, Sony Computer Science Laboratories, Japan

Proc. UIST’97

Video

ISTD 2003

Motivation : Multiple Computer Environments

• A number of (dedicated) computers within close proximity• Desktop PC’s for SW development and office tasks• Wall-size computers (displays) for collaborative

work• Personal PDA’s for mobile use

• Problems• User interface: each computer has dedicated

tethered keyboards and pointing devices, ”mouse jungle”

• Easy natural data transfer between computers

ISTD 2003

Survey of Multi-Computer Usage

• Data: 39 employees at Sony’s SW laboratories

• Q1. How many computers do you have on your desktop?3+ 54% 2 39% 1 8% 0 0%

• Q2. How often do you need to transfer data between computers on the same desktop?Very often 69% Often 25% Sometimes 3% Occasionally 0%Never 3%

• Q3. (Under situation Q2) How do you transfer data?By hand 63% Shared files 63% FTP 57% e-mail 34%Floppies 20% Other 23%

ISTD 2003

Survey of Multi-Computer Usage (cont.)

• Q4. How often do you need to transfer data from your computer to another’s computer within a short distance?

Very often 28% Often 23% Sometimes 36% Occasionally 5%

Never 5%

• Q5. (Under situation Q4) How do you transfer data?By hand 54% Shared files 57% FTP 38% e-mail 73%Floppies 11% Other 19%

Lack of easy direct data transfer user interfacessuch as copy-and-paste or drag-and-dropbetween computers

ISTD 2003

Tentative Solutions to Problems

• User interface: each computer has dedicated keyboards and pointing devices, ”mouse jungle”

More sophisticated tetherless input device such as a stylus• Can be shared between many pen sensitive displays

• Data sharing between computers ”Pick-and-Drop” direct manipulation technique

ISTD 2003

Pick-and-Drop

• Extrapolation of Drag-and-Drop technique

Drag-and-Drop vs Pick-and-Dropgrab pick updrag hold and move (virtually)drop drop

No need to drag (slide) the object on the display surface Virtual hold and move allows extended functionality,

e.g. intra-computer operations

ISTD 2003

Remote Copy vs Pick-and-Drop

Conceptual difference between remote copy and Pick-and-Drop

ISTD 2003

State Transition Diagrams

• Pick-and-Drop can coexist with Drag-and-Drop• Time-out between pen-down and pen-up

• Proximity to screen is detected by combining motion events and a time-out

ISTD 2003

System Architecture

ISTD 2003

Implementation Details

• Computers and displays• PDA’s: Mitsubishi Amity• Desktop screens: Wacom PL300• Wall-sized display: Wacom MeetingStaff +

projector

• Input device: Wacom stylus• Can distinguish at most 3 different pens

simultaneously• Probably not sufficient for practical applications• Additional pens via RF tags or IR beacons

ISTD 2003

Implementation Details (cont.)

• Computer network• Ethernet for desktops and wall-sized display• WLAN for PDA’s

• Application development in Java• Object transfers via Java serialization

ISTD 2003

Pen-ID’s

• Each pen is assigned a unique ID• ID is readable when pen is close to display• ID represented by modifier buttons in the pens

• PenManager• Binds object ID’s to pen ID’s (~ pen virtually

holds the object)• Manages object shadows• Controls data transfer

ISTD 2003

Object Shadows

(a) Pen contacts the display(b) Pen is lifted up, but is close to the display

-> if pen has data, object shadow appears to indicate this

(c) Pen is away from the screen -> no object shadow

ISTD 2003

Example Applications

Information exchange between PDA’s and kiosks

• Pick-and-Drop allows easy natural data transfer from one computer (display) to another

Transfer of data from a ”push media” information

kiosk to PDA

Transfer of data between PDA’s

ISTD 2003

Example Applications (cont.)

Drawing on a wall display with a tablet• Simple paint editor on a tablet (color, brush style)• No need for a dedicated tool-palette

ISTD 2003

Example Applications (cont.)

PaperIcons• Information exchange between a computer and a

book

ISTD 2003

Example Applications (cont.)

Anonymous Displays• Several small tablets are used as ”temporal work

buffers”• Pick-and-Drop supports intuitive data transfer without

bothering with each computer’s symbolic name

• In comparison to virtual buffers of GUI’s, physical tablets• can be arranged freely on physical desktop• provide more natural spatial interface• allow handling more than two buffers simultaneously• are not resricted by the limited and fixed size of main desktop

ISTD 2003

Discussion

Physical vs Symbolic

• Functionally, Pick-and-Drop is no more than remote copy

• However, in terms of UI, Pick-and-Drop is both physical and visible as opposed to symbolic• Supports direct manipulation of data instead of needless

exchange of symbolic concepts (e.g. drive and file names)• Supports collaborative work

ISTD 2003

Discussion (cont.)

Shared Files vs Pick-and-Drop

• Shared files force the user to deal with symbolic concepts such as machine and file names

• Difficult to keep track of multiple computers with shared files

• The unit of data transfer is not necessarily a file, e.g. text string such as URL

• Shared files a good solution for geographically separated computers, but not so intuitive between computers within close proximity

ISTD 2003

Candidate Improvements

• Support for a larger number of identifiable pens with RF tags

• Support for other file transfer protocols• Enhanced pen operations such as grouping of objects• Integration with a video conferencing system for

sharing of data of remote work spaces• Wireless mouses instead of a pens

Further developed version described in:Jun Rekimoto, ”A Multiple Device Approach for Supporting Whiteboard-based Interaction”, Proc. CHI 98.

ISTD 2003

Remarks

• Pick-and-Drop adds physical dimension to user interface• Traditional data transfer methods are too virtual

and hard to learn due to their lack of physical aspects

• Pick-and-Drop does not allow manipulating objects that are out of the user’s physical reach

• Pick-and-Drop requires a stulys-sensitive surface for operation

ISTD 2003

Augmented Surfaces: A Spatially Continuous Work Space for Hybrid

Computing Environments

Jun Rekimoto, Sony Computer Science Laboratories, Japan

Masanori Saitoh, Keio University, Japan

Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 99), Pittsburg, PA

ISTD 2003

Augmented Surfaces: A Spatially Continuous Work Space for Hybrid

Computing Environments

Jun Rekimoto, Sony Computer Science Laboratories, Japan

Masanori Saitoh, Keio University, Japan

Proc. CHI’99

Video

ISTD 2003

Motivation : Hybrid Computing Environments

• Different types of computers and devices in our everyday working (and living) environments• Portable computers• Desktops• Projectors• Digital whiteboards• Table and wall displays

• Challenges• Smooth integration of portable/personal and pre-

installed/public computers for information exchange• Connecting physical and digital (virtual) spaces

ISTD 2003

Spatially Continuous Workspace

Evolution of spatially continuous workspaces:(a) user performs individual tasks with a portable computer(b) table becomes an extension of the portable computer(c) pre-installed computer displays (table and wall) serve as

shared workspaces for collaborative tasks

ISTD 2003

Key Features of System Design

• Environmental computers as extensions of individual computers

• Supports for links between digital information and physical objects

• Spatially continuous operations(a) object dragged on computer screen(b) object ”jumps” to table, when cursor

reaches edge of the screen(c) object is dragged to the wall(d) object is linked with a physical object

ISTD 2003

Prototype Environment: InfoTable & InfoWall

ISTD 2003

System Architecture

Portable computers and physical objects have visual markers (printed 2D barcode)

ISTD 2003

Desksat

• Problem: how to cover the entire table surface with a single camera and sufficiently high resolution ?

• Solution: use a combination of two cameras• Fixed camera: monitors whole table for changed

areas• Pan/Tilt camera: scans the table in 36 (6x6)

pieces and re-visits changed areas for high resolution images

• Added/removed object is registered in few seconds

ISTD 2003

Visual Marker Recognition

• Each physical object has a visual marker (2D matrix code)

• Markers can identify 224 different objects• Smallest size of a marker is 2cm x 2cm• Recognized entities: ID, marker’s position and

orientation

ISTD 2003

Hyperdragging

(a) (b) (c)

InfoTable and InfoWall provide spatially continuous workspace (low-resolution peripheral information

space)to the laptop (high-resolution focal information

space)(a) object is dragged towards the edge of the screen(b) object (and cursor) migrates onto InfoTable(c) object (and cursor) migrates from InfoTable onto InfoWall

ISTD 2003

Anchored Cursor

• Visual feedback: line from laptop to cursor• Provided when cursor is manipulated outside the

laptop’s screen• Allows identifying cursors’ owners• Can be used to show the link between

information on the table and on the laptop

ISTD 2003

Object Aura

• Represents object’s information field (data space)• Visualizes that object has been recognized by the

system• Digital data can be attached to physical objects by

placing them on the object’s aura• If object is removed, attached data is saved on network

server• Attached data is re-displayed, when object is placed on

InfoTab´le

ISTD 2003

Shared Information Surfaces

• InfoTable and InfoWall provide shared information space for users

• InfoTable does not have ”top” or ”bottom”• ”Near” sides are determined from visual

markers• Objects are automatically rotated when brought

”near”

ISTD 2003

Object Migration

• All applications are written in Java

• Object transfer with Java’s object serilization and RMI (Remote Method Invocation)

• Supported migratable object classes• Text• Sound (voice notes)• URL’s• File short-cuts• Image files

ISTD 2003

User Experiences and Comments

• Hyperdragging easily understood and accepted• Some users found moving objects across long

distances tiresome• Allows manipulating objects that are out of the

user’s physical reach• Mapping scale between pointer movement and

the pointing device greatly affects usability• InfoTable’s 20 dpi vs laptop’s screen’s 100 dpi

• Users appreciated attaching objects onto InfoWall while sitting at the table

• Integration with e.g. printers and scanners?

ISTD 2003

More Information

Dr. Jun RekimotoDirector

Interaction LaboratorySony Computer Science Laboratories

http://www.csl.sony.co.jp/person/rekimoto.html