validation of design tools-ppt for cedar meeting-04-15-2016

CED RClemson Engineering Design Applications and Research

Varun [email protected]

Committee: Dr. Gregory Mocko (Chair)

Dr. Joshua SummersDr. George Fadel

Similarity assessment of design problemsused in creativity research

mailto:[email protected]

2 of 26

[email protected] RClemson Engineering Design Applications and Research

Research background Research objectives Design problems in creativity research Design problem network Problem similarity assessment

– Based on structural elements– Based on Latent Semantic Analysis

Conclusions Other tasks completed Future work References

Outline

3 of 26


Research Overview

Study 1

Problem 1

Method 1

Metric 1 Subject

Protocol 1

Study 2

Problem 2

Method 2

Metric 2 Subject

Protocol 2

Evaluate the impact of different methods on design creativity

- Various sources of difference between studies are present

- What prohibits comparison of two methods based on published literature alone?

- Design problems and their place in creativity research

4 of 26


Title Author, YearCognitive processes and ill-defined problems: A case study from design

Eastman, C. M., 1969

Dilemmas in a General Theory of Planning Rittel, H. W. J., & Webber, M. M., 1973

The structure of ill-structured problems Simon, H. A., 1977

A Proposed Taxonomy of Mechanical Design Problems Dixon, J. R., 1988

The structure of design problem spaces Goel, V., and Pirolli, 1992

A Suggested Taxonomy for Engineering Design Problems Frost, R. B., 1994

Mechanical Engineering Design Complexity Metrics: Size, Coupling, and Solvability

Summers, J. D., and Shah, J. J., 2010

In search of effective design problems for design research Durand et al., 2015

Research background

5 of 26


Research opportunities

Summers and Shah’s approach Durand and Linsey’s approach

Opportunitya. How to illustrate the application in

conceptual problems?b. Can natural language

representation be used to compare problems for similarity?

Opportunitya. How to assess problem similarity

based solely on problem representation?

6 of 26


Understand the pattern of design problem usage. Enable similarity comparison between conceptual design problems. Evaluate the impact of problem size on between-study treatment effects.

Research Objectives

7 of 26


Statement of requirement, needs, functions or objectives. Most problems are ill-defined and co-evolve with solutions. Conceptual design problems differ from real life design problems. Examples:

Design Problems in creativity research

Design Problem 1 (Linsey, 2012)

Design and build a low-cost, easy to manufacture peanut shelling machine that will increase the productivity of the African

peanut farmers. Target throughput is approximately 50 Kg per hour. The goals

include: a) Must remove the shell with minimal damage to peanuts b) Electrical

outlets are not available as a power source c) A large quantity of peanuts must be

quickly shelled.

Design Problem 2 (Mulet, 2012)

It is asked to design a new table for offices that allows alternate sitting and

stand up work. There are a lot of people who must work on sitting position the full day. The possibility to alternate positions

during working time could drive to an improvement in health and productivity. The current tables that allow combining

positions in work have limited surface, not enough for design, architecture and

engineering needs.

8 of 26


Problem Usage Network

ProblemAuthor/s

9 of 26


Metric Value ConclusionsNetwork density 0.0751 Poor connectivity between problems or vice-versa (between

researchers)

No. of nodes with degree 0 or 1

20* High percentage of problems which have not been re-used

No. of communities

20** - Presence of sub-groups- Problem sharing within sub-groups

Network Analysis

* Total nodes = 50** Includes isolated nodes

10 of 26


Lack of collaboration Absence of guidelines for formulating conceptual problems No ‘benchmark’ problems in practice Absence of methods to compare problems Impact of problems on creativity results not understood fully

Possible reasons for disconnectedness

11 of 26


Why compare problems?– Help reduce variability in structure of design problems– Enable problem replacement in pre-post test experiment designs– Provide metric for assessing the choice of problem– Enable use of ‘similar’ problems in practice

Methods for problem similarity assessment– Approach 1: Identification of structural elements in problem

statement– Approach 2: Latent Semantic analysis of problem statements

Objective is to compare similarity based on problem representation

Assessment of problem similarity

12 of 26


Approach 1: Element identification

DesignProblem

Goals of the problem

Functional Requirements

Non-functional requirements

Information about end user

Reference to an existing product

13 of 26


Goal of the problem– Final objective of design task– Problem may have more than one goal– Ratio variable

Functional requirements– Primary functions of designed object– Should not insinuate about method for achieving function– Generally action verbs associated with objects– Can be nouns derived from verbs (eg. washing machine)

Non-functional requirements– Do not describe primary function– Determine shape, size, operation and design selection

Approach 1: Element definitions

14 of 26


Information about end user– Knowledge of end user in problem statement– Should be explicitly stated– Categorical variable (Yes/No)

Reference to an existing product– Whether design problem hints at an existing product known to

reader.– May be a ‘conundrum’ due to cultural differences– Categorical number (Yes/No)

Approach 1: Element definitions (contd.)

15 of 26


Approach 1: Element identification example 1

16 of 26


Approach 1: Element identification example 2

17 of 26


Based on elements identified and their ‘counts’ problems can be compared

Approach 1: Comparing two problems

Design Problem

Design Problem Element Example 1 Example 2

No. of goals 1 1

No. of Functional requirements 1 2

No. of non-functional requirements 2 4

End user info. (Yes=1/No=0) 1 1

Reference to an existing product (Yes=1/No=0) 1 1

18 of 26


Definition of elements always subject to subjective interpretation Test whether people identify the same elements and their quantities. Method used:

– 4 evaluators/raters chosen– Evaluators are presented with definitions of the 5 elements– An identification example is shown to all evaluators separately– 4 evaluators asked to identify the 5 elements from 4 design

problems– Inter-rater agreement evaluated

Approach 1: Verifying similarity comparison protocol

ElementsNo. of goal

Functional requirement

Non-Functional

requirementEnd user

infoRef. to

existing product

Krippendorff’s alpha

1.0 1.0 0.863 0.184 0.598

19 of 26


Pros– Useful as a starting guide for problem formulation– Offers useful and simple mean for comparing 2 problems– Helps identify ‘information content’ of problem statements– Method shows good correlation between evaluators for most

elements– Compares problems based on representation

Cons– Subjectivity of human interpretation still involved– Cannot compare ‘knowledge content’ and skill needed in problem

solving.

Approach 1: Conclusions

20 of 26


Linguistic similarity of design problem representations Relies on extracting contextual meanings from large text corpus. Assumes that words and phrases with similar meaning are used in

similar context Generates similarity scores between -1 and 1 LSA tool available at http://lsa.colorado.edu/ used 45 design problem statements tested for LSA similarity

Approach 2: Latent Semantic Analysis

21 of 26


Approach 2: LSA results

Problem statements

Design an urban bi or tri cycle for use by white collar workers

Design a concrete mixer which can operate using a bicycle pedal mechanism.

Design a device which can compost waste vegetables.

Design a reading device for old people which can read the newspaper for them.

Design a water lifting device.

Propose alternative solution to coal pile problem at thermal plant, since the plant may not have enough land nearby to store the coal on ground.

0.02 0.01 0.06 0.01 0.03

Design of a next generation alarm clock which ensures easy operations like change of time and alarm stop unlike conventional clocks.

0.09 0.26 0.15 0.13 0.12

Design of a litter collection device for volunteers. 0.23 0.48 0.6 0.25 0.39

Redesign an electric toothbrush for increased portability. 0.01 0 -0.01 -0.01 0.03

Design alternative means to manually propel boats which are easy to maneuver, don't rock the boat or splash water.

0.09 0.15 0.15 0.06 0.52

A portion of the LSA result matrix

22 of 26


Pros– Objective way of comparing two problem statements– Based on problem representation only– Backed by statistical computation, avoids subjectivity– Can detect contextual changes between text bodies even with slight

modifications (viz. design vs redesign etc.)

Cons– Results only as good as the corpus of text used– Based solely on representation, problem ‘solvability’ can’t be

compared

Approach 2: Conclusions

23 of 26


Evaluate the impact of problem size on between-study treatment effects.– Meta-Analysis of different studies to assess treatment effect.– Analysis of heterogeneity between treatment effect of various

studies– Meta-regression approach to understand impact of problem size on

heterogeneity

Other works completed

24 of 26


Write ThesisDefend

Enjoy Summer

Future work

25 of 26


Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., and Markman, a. B., 2011, “An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods,” J. Mech. Des., 133(3), p. 031008.

Chulvi, V., Sonseca, A., Mulet, E., and Chakrabarti, A., 2012, “Assessment of the Relationships Among Design Methods, Design Activities, and Creativity,” J. Mech. Des., 134(11), p. 111004.

Ameri, F., Summers, J. D., Mocko, G. M., and Porter, M., 2008, “Engineering design complexity: An investigation of methods and measures,” Res. Eng. Des., 19(2-3), pp. 161–179.

Rittel, H. W. J., and Webber, M. M., 1973, “Dilemmas in a General Theory of Planning,” Policy Sci., 4(December 1969), pp. 155–169.

Simon, H. A., 1977, “The structure of ill-structured problems,” Models of discovery, Springer, pp. 304–325.

Dixon, J. R., Duffey, M. R., Irani, R. K., Meunier, K. L., Orelup, M. F., 1988, “A Proposed Taxonomy of Mechanical Design Problems,” Comput. Eng., 1, pp. 41–46.

Goel, V., and Pirolli, P., 1989, “Motivating the notion of generic design within information-processing theory: the design problem space,” AI Mag., 10(1), pp. 18–36.

Goel, V., and Pirolli, P., 1992, “The structure of design problem spaces,” Cogn. Sci., 16(3), pp. 395–429.

Frost, R. B., 1994, “A Suggested Taxonomy for Engineering Design Problems,” J. Eng. Des., 5(4), pp. 399–410.

Summers, J. D., and Shah, J. J., 2010, “Mechanical Engineering Design Complexity Metrics: Size, Coupling, and Solvability,” J. Mech. Des., 132(2), p. 021004.

Tsenn, J., Helms, M. E., Linsey, J. S., and Mcadams, D. A., 2015, “In search of effective design problems for design research,” pp. 1–10

References

CED RClemson Engineering Design Applications and Research

Questions?

27 of 26


Appendix A: Design problem used for experimental verification of protocol

Problem 1: Design a new system for gathering together and hiding the wires of the electronic equipment in an office table. Currently the work in the field of design, architecture and engineering needs of a personal computer, printers, and scanners. Each of these devices needs of electrical supply and the wires on table surface are annoying. Actually, there are simple solutions to gather them, but it is difficult to extract or introduce a wire, or they leave the wires hanging behind the table.

Problem 2: Design an automatic recycler device that can automatically sort plastic bottles, glass containers,

aluminum cans, and tin cans. The major differentiation between types of materials lay with the given dimensions of the products: plastic bottles are the tallest, glass containers are very short and heavy, and aluminum cans are lightweight. Devices are given strict requirements to adhere to such as volume and weight constraints, safety requirements, and most importantly, have to operate autonomously once a master shutoff switch is toggled.

Problem 3: A mechanical system is required which, in the event of a fire, will enable people to escape from a

six-storey building by lowering themselves to the ground from windows. The system, which might make use of a 5 mm diameter steel cable, must be capable of lowering either a small child or a heavy adult at approximately the same constant speed.

Problem 4: Design and develop an artifact to facilitate grocery shopping in a typical French/Singapore

Chinatown fresh market. The artifact should facilitate carrying of groceries from fresh market to home.

28 of 26


S.No. Characteristic Definition Scoring system

1. Reference to existing product

Whether or not the problem statement contains a reference to an existing product.

Questions to be asked Does the problem statement contain reference to

an existing product? Does the problem statement ask you to re-design

an existing product?

If a reference to existing product exists in the problem statement, assign a score of 1, else 0

2. Functional requirements (FR)

These are the things which the product needs to do, or the tasks that you want the product to perform without any reference to how it should be done. Judgement should be based only on explicitly stated texts, and not on the implied meaning of a sentence. To identify FRs, look for:1. Action verbs like move, work etc. associated with

objects (objects include nouns on which the action verbs act like throw stones, gather fruits etc.)

2. Primary functions of the design (eg. move objects, lift, transport etc.).

3. These could also be nouns derived from verbs (eg. washing machine, toaster etc.)

4. If there are two objects associated with one primary function, count it as 2 separate FRs (eg. move object X & object Y is counted as 2 FRs)

Questions to be asked What are the primary functions of the product? What are the expected outputs/tasks which the

product needs to perform?

Count the number of functional requirements given in the problem statement. There can be 2 cases:

1: When a new product design is desired: In this case, FR should be specified in the problem statement. Else, give a score of 0.

2: When a re-design or a new design for an existing product is desired: FR count in this case is already 1 to start off, since atleast 1 product function is known.

Appendix B: Element definitions

29 of 26


S.No. Characteristic Definition Scoring system

3. Non-functional requirements

These are 'non-functional' requirements, which do not determine the primary functions of the product, but cast a bound on the overall shape, size, cost, operation and selection of the design. Judgement should be based only on explicitly stated texts, and not on the implied meaning of a sentence. Typical NFRs include:1. Any restrictions on what the product or system shall or shall

not do apart from the primary functions (eg. should not overheat, should be easy to use, should be manufacturable etc.)

2. Any restriction on how the product shall fulfil its intended functions (eg. device should move using rollers, device should work using sliding mechanism etc.)

3. Any qualities that the product must possess (eg. easy to make, easy to use, cheap, etc.)

Questions to be asked• What things cast a limit or a bound on the solution space? What are the qualities that the product should possess as a

whole? How the overall product 'shall be' like?

Count the number of Non-functional requirements given in the problem statement.

4. Number of goals

These are goals or final objectives associated with the design problem.

Questions to be asked What is the final objective of the problem statement, or Does the problem statement ask only to design or do

something else?

Count the number of goals or objectives mentioned in the problem statement.

5. Information about end user

Information about who is going to use the product or who is the customer. It should be explicitly stated in the problem statement.

Questions to be asked Who is going to be the end user of the product?

Check the problem statement to see if any information about the end user is provided or not. If yes, give a score of 1, else a 0

Appendix B: Element definitions

30 of 26


AuthorsEastman(1969)

Rittel and Webber (1973)

Dixon(1988)

Goel(1992)

Frost(1994)

Summers and Shah (2010)

Durand and Linsey (2015)

Lack of well -defined specifications for goals

Lack of formal representation language

Lack of definitive formulation

Lack of known solution states

Lack of immediate tests for solutions

Lack of objectivity in solution selection

Lack of learning opportunities by trial and error

Lack of exhaustive set of solutions

Perceived need

Function

Phenomenon

Embodiment

Artefact type

Artefact instance

Lack of problem structuring

Presence of distinct problem solving phases

Reversal of direction of transformation functions

Modular structure

Incremental solution transformation

Personalized stopping rules

Abstraction hierarchies

Type of entity being designed

Degree of innovation involved

Extent of possible decomposition of designed entity

Availability of adaptable solutions

Complexity of designed entity

Degree of interaction within solution

Looseness or tightness of constraints

Number of artefacts to be built

Number of independent design variables

Number of dependent design variables

Number of design relations

Number of measures of goodness

Number of functional requirements and design parameters

Number of design constraints

Size

Connectedness

Participant's familiarity with problem

Participant's familiarity with solution and principles

Size of potential solution space

Assumed constraints due to known solutions

Effort required

Fixation

Problem domain

31 of 26


DP1 Toy to transport basketDP2 Subway improvementDP3 New drawing tableDP4 Tubular map case

DP5System to collect and hide electronic wires

DP6 New table for officesDP7 Rover deviceDP8 Automatic recyclerDP9 Wearable binocularsDP10 Bi/tri-cycle

DP11Bicycled Concrete Mixer

DP12 Vegetable ComposterDP13 Reading DeviceDP14 Water lifting deviceDP15 Traffic light using LED

32 of 26


DP16 Counting and packaging deviceDP17 Milk frothing deviceDP18 Peanut shelling machineDP19 Automatic casting systemDP20 Robotic vacuumDP21 Portable washing machineDP22 Solar heating and cooking deviceDP23 Rigless abandonment toolDP24 Wheelchair simulatorDP25 Counter top standDP26 Split pin designDP27 Device to transport ping pong ballDP28 Tool for alien speciesDP29 Biomass based cooking systemDP30 System for UAVsDP31 Remote controllerDP32 Doodle toaster

33 of 26


DP33Combined toaster and coffee maker

DP34 Horizontal toasterDP35 Crumb tray toasterDP36 Shopping cartDP37 Outdoor customer productDP38 Manual boat propulsion deviceDP39 Plastic and paper sorting deviceDP40 Lunar dust protection deviceDP41 Desk elevatorDP42 Ship guitarDP43 Bottle cap regiser machineDP44 Soda drink makerDP45 Concept to center mating partsDP46 Concept to use snow as insulatorDP47 Coal pile solutionDP48 Next gen alarm clockDP49 Litter control deviceDP50 Electric tooth brush

validation of design tools-ppt for cedar meeting-04-15-2016

Documents