designing effective step-by-step assembly instructions maneesh agrawala, doantam phan, julie heiser,...

Designing Effective Step-By-Step Assembly Instructions

Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker,

Jeff Klingner, Pat Hanrahan and Barbara Tversky

Microsoft Research and Stanford University

1 2 3

4 5 6

Designing Instructions

Planning Choose sequence of assembly

operations Robotics / AI / Mechanical Engineering

[Wolter 89], [de Mello 91], [Wilson 92], [Romney 95]

Presentation Visually convey assembly operations Visualization / Computer Graphics

[Seligmann 91], [Rist 94], [Butz 97], [Strothotte 98]

We jointly optimize plan and presentation

Geometric Analysis [Romney 95]

A

B A blocked by B

B blocked by A

both parts free to move

A B

B A

CA

B

A C

B

A C

B A C

B

Input Parts Blocking Graph

Geometric Assembly Planning

Valid Valid Invalid

Many Geometrically Valid Sequences

Valid Valid ValidValidValid

How do we choose the best sequence?

Our Approach

Identify cognitive design principles How people conceive of 3D assemblies

How people comprehend visual instructions

Encode principles as constraints within automated design system

Identifying Design Principles

Experiment 1: Assemble and draw instructions

Experiment 2: Rate effectiveness

Experiment 3: Validate effectiveness

Sequence the Assembly Operations

Single exploded view diagramStep-by-step diagrams

Step-by-step instructions preferable

Illustrate the Assembly Operations

Action diagrams preferable

Structural diagrams

Action diagrams

Ensure Visibility of Parts

Essential Parts being attached Context (show earlier parts)

Less important All parts in a symmetric group Repetitive operations

Automated Instruction Design

Step-by-Step

Action diagrams

Good visibility

TV stand instructions generated by our system

Input

Geometry Parts in assembled positions

Orientations Default viewpoint / orientation Preferred orientation for each part

Groupings fasteners, significant parts, symmetry,

similar-actions Ordering constraints

Force one part to attach before another

requiredoptional

TV Stand Input

Geometry: Parts in assembled positions

Orientations: Default viewpoint / orientation

Groupings: fasteners, significant parts, symmetry

requiredoptional

Reorientation

Sequence Parts

All parts

Best subset of parts

Leftover parts

Sequence of assembly steps

Search

Action Diagrams

Structural Diagrams

Reorientation

Sequence Parts

All parts

Best subset of parts

Leftover parts

Sequence of assembly steps

Search

All parts

...…

Part subsets

All parts

Best subset of parts

Leftover parts

Search

Reorientation

Sequence Parts

Sequence of assembly steps

All parts

Best subset of parts

Leftover parts

Search

Reorientation

Sequence Parts

Sequence of assembly steps

Interference

Attachment

Ordering

Grouping

Visibility

SearchChoose part subset

Computing Visibility

Area(P) = # red pixels

Vis(P,Q) = Area(P,Q) / Area(P)

Area(P,Q) = # red pixelsArea of top not occluded by sides Area of top alone

% pixels that remain visible

Visibility Constraint

Current parts R min ( Vis(r, R-r) )

Check that each part in current subset is visible

Attached parts A Vis(A, R)

Check that context is visible

Unattached parts U min ( Vis(u, R) )

Check that future parts will be visible

r R

u U

Lego Car

Input model

Bookcase

Input model

Add significant parts one by one

Visibility

Distance to viewer

Add all remaining parts

Omit repetitive operations

Skip if 2 similar-action parts already added

Sequence Parts

Best subset of parts

All parts

Leftover parts

Search

Reorientation

Sequence of assembly steps

Bookcase – After Sequencing

Bookcase – Omitting Repetition

Set preferred orientation for significant parts

If visibility of current parts is low try alternate oblique views

Sequence Parts

Best subset of parts

All parts

Leftover parts

Search

Reorientation

Sequence of assembly steps

Bookcase – With Reorientation

Reorientation

Sequence Parts

All parts

Best subset of parts

Leftover parts

Sequence of assembly steps

Search

Action Diagrams

Choose Direction

Build Stacks

Place Guidelines

Reorientation

Sequence Parts

All parts

Best subset of parts

Leftover parts

Sequence of assembly steps

Search

Sequence of assembly diagrams

Building Stacks

Stack: set of parts that Share separation direction Lie on a stabbing line Are in sequential contact

Base

1st stack part

2nd stack part

3rd stack part

4th stack part

Base

1st stack part

Base

1st stack part

Building Stacks

Stack: set of parts that Share separation direction Lie on a stabbing line Are in sequential contact

Base

1st stack part

Placing Guidelines

Expand stacks Form lines between stack parts

Placing Guidelines

Expand stacks Form lines between stack parts

Bookcase

9 parts Planning: 48s

Table

13 parts Planning: 28s

Test Object

25 parts Planning: 53s

Exploded View

Future Work

Experiment 4: Evaluation Time/errors as they use our instructions

Assemblies with more subparts Take advantage of hierarchy Apply system recursively

Physical and functional constraints Part size and mass Gravitational stability

Summary

Cognitive design principles Step-by-step Action diagrams Good visibility

Automated instruction design system

Integrate planning and presentation

Acknowledgements

Boris Yamrom

Christina Vincent

ONR grants N000140210534, N000140110717 and N000140010649