A Design Data Analysis Approach to Early Stage Design Process

NSF Innovation and Discovery Workshop: The Scientific Basis of Individual and Team Innovation and Discovery

May 18, 2006

Maria C. YangUniversity of Southern California

Under the support of NSF DMI-0547629

Early stage design processHow are things designed?

• High-impact phase • Design practice and design teacher• Good design process linked to good design

outcome– Deeper understanding of design methodologies

• Examine output of design activities over time

[Ulrich & Eppinger 95; Pahl & Beitz 96].

Research questions

• What is “good” early stage design process? • What is “good” design outcome? • How are process and outcome linked?• How can process metrics help design?

• Research goals:– Comprehensive measures of early design process– Time-based models of process information– Framework that links measures with outcome ->

indicators

Research framework

Early stage design processDesigners

Design problem

Final concept

Design context: Process measures

Design content: Concepts

Design data

Controls: Resources & strategy

Clarify Generate Select

(1)

(2)

(3)

Research framework

Early stage design processDesigners

Design problem

Final concept

Design context: Process measures

Design content: Concepts

Design data

Controls: Resources & strategy

Clarify Generate Select

(1)

(2)

(3)

Concept generation and sketching• Fluency, flexibility, originality• “Quantity breeds quality” [Osborn]

– IDEO approach [Kelley & Littman]

• Sketching tied to design cognition [Nagai & Noguchi 03; Suwa & Tversky 97; Goel 95; Cross; Shah; Goldschmidt; Tovey]

• Questions– How might concept quantity be linked

to design outcome?– Sketch quantity?– If sketching is a design language, does

drawing skill impact design performance? [Yang & Cham (accepted); Cham & Yang 05]

Research/Teaching Test bed

• Design courses at Caltech– Engineering design contest (2 different years)

– Introduction to design

• Outcomes– Grades

– Contest performance (Engineering Contest)

Related work

• Sketch classification– Function [Ullman 90; Ferguson

92; van der Lugt 05; Goel 95]

– Elements [McGown 98; Rodgers 00]

• Sketching and outcome– Teams who sketch vs.

those who don’t [Schütze 03] – 3D sketching & outcome

[Song & Agogino 04]

• What about sketching skill?

1. Concept generation

• Correlations between brainstormed ideas and grade– Text lists and sketches

• Introduction to Design course only• Beginning of project

Correlation Coefficient, Rs

Project grade Final term grade

Total ideas brainstormed

0.48 0.33

N = 33, Rs = 0.291 for = 0.05

2. Sketch quantity and design outcome

N = 24, Rs > 0.343 for α = 0.05

N = 21, Rs = 0.370 for = 0.05

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1 2 3 4 5 6 7 8 9

Week

Correlation Coefficient, Rs

Rs with grade totalsketches

Rs with grade Dimen-sioned

Rs with contest totalsketches

Rs with contest Dimen-sioned

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3 4 5 6 7 8 9

Week

Correlation Coefficient, Rs

-Design earlier vs. later-Engineering DesignContest

Year 1

Year 2

3. Sketching Skill

a. What is the nature of sketching skill in design? – Generic or task-based?– Research in mental imagery [Kosslyn]

1. Comprehensive, generic “trait” 2. Task-based skill3. Somewhere between 1) and 2)

– Hypothesis• Sketching ability similar to (3)

b. How is skill related to design outcome?– Hypothesis: Sketching skill important, but not only

factor

Results: 3a. Types of sketching skill

• Possible results1. Comprehensive skill: Strong correlations between tasks2. Task-based skill: No correlation3. Skill lies between the two: Range of correlations

• Results suggest option 3

Correlation between sketch tasks. N = 32, Rs >= 0.350 for = 0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Bike task and Handtask

Bike task and Boxtask

Hand task and Boxtask

Correlation coefficient, Rs

3b. Sketching and Design Outcome

• Sketching skill: No clear trends• Design process depends on many skills/factors• Project type, outcome measures• More studies needed

N = 32, Rs >= 0.350 for = 0.05

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Project grade Class grade Avg. ranking

Correlation coefficient, Rs

Bike task

Hand task

Box task

Thoughts on future work

Early stage design processDesigners

Design problem

Final concept

Design context: Process measures

Design content: Concepts

Design data

Controls: Resources & strategy

Clarify Generate Select

(1)

(2)

(3)

Research questions 2 & 3

2. How is sketching ability linked to fluency?– Hypothesis: Those who draw better also draw

more

2. Sketching ability and sketch fluency

• Total: Drawing “well” correlates positively • 3D: Bike task correlates negatively• Drawing skill vs. other means of visualization?

N = 32, Rs >= 0.296 for = 0.10

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Bike task Hand task Box task

Correlation Coefficient, Rs

Total sketches

3D sketches only

Concluding remarks

• What does this say about sketching skill in engineering design? – Sketching skills not created equal– Gearheads and Artists

• How is sketching ability linked to fluency? – Maybe - ability to visualize without drawing– Logbook resisters

• Does better sketching also mean better design? – No clear trends – Design requires many skills, sketching only one

– Outcome measure consistency

Conclusions

• Quantity may correlate with quality, early on

• In course 1, last minute sketching correlates negatively

• Dimensioned drawings– Linked closely to prototyping– Would expect it better to delay– Class heavily emphasizes prototyping

Research in early stage design

• Clarifying design requirements– Need finding [Faste 87], Voice of Customer [Griffin & Hauser

93], QFD [Hauser & Clausing 88]

• Concept generation & creativity– Brainstorming [Osborn 63], Deep Dive [Kelley & Littman 01],

Conceptual blockbusting [Adams 76], TRIZ [Altshuller 99]; shape grammars [Schmidt & Cagan 97], Method of Imprecision [Wood & Antonsson 89]

• Concept selection– Concept selection [Pugh 91], optimal design [Papalambros &

Wilde 88], robust design [Phadke 89]

• Descriptive approaches– Protocol studies [Blessing 95; Bucciarelli 94; Tang & Leifer 91]– Text analysis [Mabogunje & Leifer 96; Dong & Agogino 97]– Sketch analysis [McKim 80; Schön & Wiggins 92; Ullman 90]

• (does not incl work in other fields: cog sci, psych, ….)

Sketching results

Engineering design contest 1 Engineering design contest 2

• Average weekly total and dimensioned sketches• Proportionally fewer dimensioned early on• Conceptualization earlier, prototyping later• Peak sketching for (1) earlier than for (2)

Role of design activities

• Concept generation [Yang 03]

– “Going for quantity”

• Prototyping & time [Yang 05, Yang 04]

Design outcome measures

• Spearman Rank Correlation

2

13

61

N

ii

s

dR

N N=

⋅∑= −

−

Rs = correlation coefficientdi = Xi – Yi where X and Y are ordinal rank of the variablesN = sample size

Rs value between –1 and 1If –1 < Rs < 0, negative correlationIf 0 > Rs > 1, positive correlationIf Rs > threshold, statistically significant correlation

Survey to assess drawing skill(do try this at home)

1. Draw a bicycle from memory2. Draw your non-dominant hand holding

two small objects3. Draw a rectangular box that is open

at the top. Inside the box is a rubber ball. The front of the box has a large button, and each side of the box has a large “X” painted on it.

3b. Sketching and Design Outcome

• Sketch fluency: Positive but no sig. correlation• Sketching skill: No clear trends• Design process depends on many skills/factors• Project type, outcome measures• More studies needed

N = 33, Rs >= 0.291 for = 0.10 N = 32, Rs >= 0.296 for = 0.10

0

0.05

0.1

0.15

0.2

0.25

0.3

Project grade Class grade Avg. ranking

Correlation coefficient, Rs

Total sketches

3D sketches only

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Project grade Class grade Avg. ranking

Correlation coefficient, Rs

Bike task

Hand task

Box task