Magic Cards: A Paper Tag Interface

for Implicit Robot Control

Shengdong Zhao Takeo IgarashiKoichi Nakamura Kentaro Ishii

JSTERATO

Average Time on Housework Per Week

Source: National Survey of Families and Households

MarriedDivorced

SeparatedWidowed

Never Married

Women 29.83 hrs 27.80 hrs 21.17 hrs

Men 17.67 hrs 21.53 hrs 17.83 hrs

Domestic Robots can Help, but …

Technology: Robots technologies are not well defined, and may change often.

User: Housewives and elderly are afraid of technologies

Environment: Robots may disturb the comfort of our home

Domestic Robots can Help, but …

Technology: Robots technologies are not well defined, and may change often.

User: Housewives and elderly are afraid of technologies

Environment: Robots may disturb the comfort of our home

Domestic Robots can Help, but …

Technology: Robots technologies are not well defined, and may change often.

User: Housewives and elderly are afraid of technologies

Environment: Robots may disturb the comfort of our home

Domestic Robots can Help, but …

Technology: Robots technologies are not well defined, and may change often.

User: Housewives and elderly are afraid of technologies

Environment: Robots may disturb the comfort of our home

Domestic Robots can Help, but …

Technology: Robots technologies are not well defined, and may change often.

User: Housewives and elderly are afraid of technologies

Environment: Robots may disturb the comfort of our home

1

2

3

1. Frequent Change of Robot Technology

Principle of Information Hiding– Hiding "difficult design decisions or design

decisions which are likely to change“ (David Parnas, 1972)

Housework tasks

Robot technology

(Long historyrelatively stable)

(Change frequently)

Interface

Human users

1. Frequent Change of Robot Technology

Principle of Information Hiding– Hiding "difficult design decisions or design

decisions which are likely to change“ (David Parnas, 1972)

Housework tasks

Robot technology

Interface

Human users

Solution: Only expose “tasks” to users

2. Fear of Learning New Technology

Everyday Objects (Such as Paper Cards)

Solution: Interact with everyday objects

New Technology

???

3. Possible Disturbance at Home

The Elves and the Shoemaker (Fairy Tales, by the Grimm Brothers)

Oyster Girl(A Chinese Fairy Tale)

Solution: Use asynchronous interaction

Magic Cards Video

Guest Star: C3PO

The Robot Housework System

Human users

Magic Cards

Interface

Sensors andProcessing System

Robot Housework System

computer

cameras

5m

4m

robotscomputer

Vision-based Environment

Robots Used

Robots Used

Robots Used

Robots Used

The Robot Housework System

Human users

Magic Cards

Interface

Sensors andProcessing System

Robot Housework System

The Magic Cards Interface

Magic Cards is a very simple tangible programming interface for housework planning using paper cards.

The Magic Cards Interface

Magic Cards is a very simple tangible programming interface for housework planning using paper cards.

Grammar of the Magic Cards Task Planning

Language

C3PO cleans the dishes at noon.

Grammar of the Magic Cards Task Planning

Language

C3PO cleans the dishes at noon.

subject + verb + object + modifiers

The dishes need to be cleaned at noon. object + verb + modifiers

Grammar of the Magic Cards Task Planning

Language

The dishes need to be cleaned at noon. object + verb + modifiers

Action tagsObject tags Modifier tags

Object Tags

Action Tags

Vacuum

Mop Deliver

Washdishes

Washclothes

Avoid

…

Ideally include all types of house work tasks

Supported Action Tags

Vacuum

Mop Deliver

Avoid

Action Tags: Vacuum, Mop, Avoid

Vacuum

Mop Avoid

Action Tags: One to One Delivery

Action Tags: Many to One Deliver

Action Tags: One to Many Delivery

Trash binDoor

Modifier Tags

Time Tags

Order Tags

Routine Tasks1. Move the sofa away2. Vacuum underneath3. Move the sofa back

Routine Tasks1. Move the sofa away2. Vacuum underneath3. Move the sofa back

Routine Tasks1. Move the sofa away2. Vacuum underneath3. Move the sofa back

Routine Tasks1. Move the sofa away2. Vacuum underneath3. Move the sofa back

1

1

Routine Tasks1. Move the sofa away2. Vacuum underneath3. Move the sofa back

1

1

2

Routine Tasks1. Move the sofa away2. Vacuum underneath3. Move the sofa back

Compound TasksDefine a function:

CompoundTask C1 (object) {Move object away;Vacuum underneath;Move object back;

}

Use it sometime later:C1(anotherObject);

Challenge: How to incorporate this concept into Magic Cards so that non-tech-savvy people can program compound tasks?

Special Purpose Tags: Compound Task Tag

Special Purpose Tags: Compound Task Tag

Clean underneat

h

Evaluations1. Iterative design of the Magic Cards

interface

2. User study of both the Magic Cards interface and the prototype system

Iterative Design of Magic Cards

Goal– understand users – receive initial feedback

on design concepts– refine paper-tag

interface

Approach– 6 participants, 2 hrs per

person (1 male, 5 females, age range 21-64, mean 35.8)

– Semi-structured interviews

Iterations

Action Tags

Time Tags

Modifier Tags

Order Tags

Special Purpose and Object Tags

User StudyGoal

– Test the “Magic Cards” interface and the prototype system

Approach– Participants:

• 8 participants, 2 hrs per person (all females, 21-52, mean 31.5)

– Procedure:1. Questionnaire2. Paper-tag evaluation3. Task scenarios in simulated

environment

Positive Results and Feedback

6/8 interpret all tags correctly

Short task planning time (within minutes)

Magic Cards is easy to understand and use– e.g., a 52 year-old lady who self-describe as

being afraid of “any buttons” highly praised the simplicity of the interface

Concerns and Suggestions

• Design for young children– Child safe– Collect cards after

planning

• All-encompassing robots preferred for small home

• Privacy and aesthetics issues of sensors– Sensors on robots

instead of in the environment?

ConclusionIntroduced an alternative HRI design

approach by focusing on “tasks”

Designed the Magic Cards interface

Developed a prototype system to support interaction with the Magic Cards interface

Future WorkExtend the Magic Cards interface with more

actions and modifiers tags

Improve the sensor and system technologies

Use more advanced robots

Washdishe

s

Washcloth

es

…

AcknowledgmentsMinghui Sun, Yoshiki Takeoka, Sorahiko

Nukatani, Tak Miyake, Zoey Yu, Yotam Gingold, Koji

Yatani,Daisuke Sakamoto

JST ERATO: http://www.designinterface.jp (Many openings for post-docs and interns!)

User study participantsAnonymous reviewers

End