evaluation of complex systems - iit institute of design · charles l. owen evaluation of complex...

Evaluation of Complex Systems

Charles L. Owen

Institute of DesignILLINOIS INSTITUTE OF TECHNOLOGY

Foun

ded

as th

e Ne

w Ba

uhau

s in

193

7, C

hica

go’s

Inst

itute

of D

esig

n, II

Tis

a c

ente

r for

adv

ance

d st

udy

in h

uman

-cen

tere

d in

nova

tion.

Le

arn

mor

e: w

ww.id

.iit.e

du

January, 2007

2 Charles L. Owen Evaluation of Complex Systems

A paper published by:

Design Studies The international journal for design researchin engineering, architecture, products and systemsVolume 28, No. 1 (2007): 73-101

Keywords: evaluation, planning, modeling, structured planning,systems design

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

Evalu

ation

of C

omple

x Sy

stems

/ Cha

rles

L. Ow

en /

Janu

ary,

2007

Charles L. Owen Evaluation of Complex Systems 3

Abstract

A new process is introduced forevaluating systems. Criteria arerepresented as a hierarchy offunctions or, at a policy level,a hierarchy of policy positions.The system to be evaluated isrepresented as a hierarchy ofits components. Elements of thesystem are scored against cri-teria using a bidirectional scalewith provision for degrees ofboth support and obstruction.Assessment values for lowest-level elements are calculatedusing a series algorithm thatrelates composite scoresuniquely to the scoring scale.Values for categories withinboth hierarchies are calculatedfrom element values belowthem, with results presentedgraphically, in color.

Introduction

Simple things are difficult enough to evaluate, even those forwhich hard data can be found. For complex entities, especiallywhen subjective evaluations are involved, the problem can bedauntingly difficult—enough so that comprehensive assessmentsoften are just not attempted.

The purpose of this paper is to present a means that combinesanalysis, modeling and evaluation to assess overall as well as de-tailed system performance against the needs of system users. Theprocess is comprehensive and is a natural extension of the analyti-cal techniques of Structured Planning, a process developed for ad-vanced planning (Owen, 1992, 1993, 2001a, 2001b, 2005).

Conventional System Evaluation ModelsSince system evaluation is admittedly complex, it would be goodto have an example system to work with that is familiar enoughthat the discussion can focus on the process without additional ex-planation of the content.

An example that meets this criterion is the restaurant. Restau-rants are complex, are frequent subjects for evaluation, and arewithin the personal experience of nearly everyone. Evaluations ofthem are ubiquitous in travel guides and every city’s newspapersand magazines. A restaurant’s workings are reasonably familiar toall.

The star model. Travel guides employ the ultimate summaryevaluation model: a number of stars or similar symbols. Miche-lin’s system is probably the most prestigious and best known ofthese. A single Michelin star means a restaurant is "a very goodrestaurant in its category"; two stars means it has "excellent cook-ing, worth a detour"; and three stars (very difficult to attain) indi-cate "exceptional cuisine, worth a special trip". Michelin limits itsstars to a maximum of three; other evaluators use four or evenfive, presumably because they can judge with greater resolution.

How the ranking is done is seldom made explicit. Pat Bruno,restaurant critic of the Chicago Sun-Times newspaper comments,"Restaurant owners all too often think it’s all about the food. Not.It’s a lot more than that. It’s the whole package—food, quality,price, service, enjoyment—that provides the gift that keeps on giv-ing, which in the restaurant business translates to steady business"(Bruno, 2004). Critics, usually traveling incognito, bring their ex-perience to bear in a way that integrates many factors in a subjec-tive, but well informed judgment. The process requires highlyrefined expertise developed over time and, of necessity, is con-fined to a narrow range of content.

The Multiple Category Model. Restaurant critics who reporton television shows or in articles on food frequently divide theirevaluations into two or three major categories. The categories usu-ally cover quality of food, excellence of service and, possibly, theambiance of the environment (which may include the niceties ofinterior design). James Ward, food critic of Chicago’s local ABCTV affiliate, organizes his personal appraisals under the terms

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


1. Food Quality

2. Presentation

3. Attentiveness

4. Knowledgeability

5. Interior Design

6. Lighting

10

5

7

6

4

6

7

4

2

4

3

3

Criteria Wei

ght

Sca

le

1 - 10

1 - 5

1 - 3

1 - 5

1 - 5

1 - 4

Sca

le

Fact

or

1

2

3.3

2

2

2.5

Minimum Values for anExceptional Restaurant

Values for Restaurantunder Evaluation

8

3

3

3

3

4

(10)(1)(9) = 90

(5)(2)(5) = 50

(7)(3.3)(3) = 70

(6)(2)(5) = 60

(4)(2)(5) = 40

(6)(2.5)(4) = 60

Minimum Values for anExcellent Restaurant

(10)(1)(8) = 80

(5)(2)(4) = 40

(7)(3.3)(3) = 70

(6)(2)(4) = 48

(4)(2)(4) = 32

(6)(2.5)(4) = 60

Minimum Values for aVery Good Restaurant

(10)(1)(7) = 70

(5)(2)(4) = 40

(7)(3.3)(2) = 46.7

(6)(2)(4) = 48

(4)(2)(3) = 24

(6)(2.5)(3) = 45

8

4

3

4

4

4

(10)(1)(8) = 80

(5)(2)(3) = 30

(7)(3.3)(3) = 70

(6)(2)(3) = 36

(4)(2)(3) = 24

(6)(2.5)(4) = 60

273.7 300330370

9

5

3

5

5

4

Min.Values Calculations Calculations Calculations Calculations

Sum = Sum = Sum = Sum =38

Sum of Weights = Minimum for

Exceptional = 37038

Range for Exceptional = 9.7 to 10

Minimum for Excellent = 330

38

Range for Excellent = 8.7 to 9.7

Minimum for Very Good = 273.7

38

Range for Very Good = 7.2 to 8.7

Restaurant Score = 300

38

MeritIndex

= 9.7 = 8.7 = 7.2 = 7.9

= 7.9

Figure 1 Criterion Function Method for Calculating a Merit Index

Min.Values

Min.Values

ActualValues

"breads" and "circuses". The Zagat Survey accu-mulates votes from volunteer members of thedining public using a 1–30 scale and three cate-gories (food, decor and service). Results are av-eraged for each category and published annuallyin books specialized for individual cities. Formore detail, the In-Room City Guide of SanFrancisco rates restaurants using five categories:(1) decor, ambiance and service; (2) scope of themenu, quality of ingredients, execution of dishes,and how dishes please the palate; (3) cuisine—classic or cutting edge, traditional or fusion; (4)wine list—geographic scope, variety, and prices;and (5), cost of dining—splurge or "casual" af-fordable.

Ratings in this assessment model are madesubjectively, much as is done with the starmodel. The advantage is that judgments can benarrowed to specific topics, and difficult integra-tion problems can be avoided—such as canoccur when the quality of food is excellent, butservice unacceptably bad. The separate judg-ments are passed on to the reader, along with theintegration task.

A variant on this model is the "filled circle"version used by institutions such as the magazineConsumer Reports. In this version, criteria arerepresented by circles that can be graphicallytreated in different ways to indicate degrees ofquality. Ratings can be directly qualitative or can

translate quantitative information to qualitative(translating, for example, price to high, medium,low). A typical, 5-value scale might have a fullfilled red circle for highest quality, a half-filledred circle for excellent quality, an open circle forgood quality, a half-filled black circle for poorquality, and a full-filled black circle for unac-ceptably bad quality.

The Criterion Function Model. Extendingthe multiple category model, a technique devel-oped in engineering design and decision sciencecan be used both to consider more categories andto integrate results (see Cross, 2000; Saaty andVargas, 2000; Pahl and Beitz, 1996; Roozenburgand Eckels, 1995; Ostrofsky, 1977; and Wood-son, 1966 among others). In this model, scalescan be individually tailored to each criterion (orobjective) and weights can be applied to enablethe composite score to reflect the relative impor-tance placed on each criterion by the evaluator oruser of the data. Differences in scales can be ac-commodated by scale factors that convert ratingsto a common scale. The final integrated score isoften referred to as a merit index.

An example will show the advantages of thismodel in basic form (Figure 1). For this exam-ple, the three classic categories of restaurantcriteria—food, service and ambiance—are ex-tended to six: food quality, presentation, atten-tiveness, knowledgeability, interior design and

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


lighting. Any number of criteria, of course, couldbe accommodated.

In the first column on the left, weights areapplied to the six criteria based on the judgmentof the evaluator as to how important each is tothe overall rating. Scales for rating can be indi-vidually specified, and this is done for each cri-terion in the second column. In the third column,scale factors are given to transform the scales toa common range, 1 to 10 in this example. Thescale factors allow criteria to be judged on scalesmost appropriate to their qualities while prevent-ing unfair calculation advantages due only to dif-ferences in the scales of evaluation.

In the next three paired columns, values aregiven and calculations made for minimum criter-ia thresholds for each of three generally recog-nized quality standards: exceptional, excellentand very good. Each calculation multipliesweight by scale factor by value. The evaluatormust decide the minimum value that the criterioncould achieve and still be awarded the qualitydistinction under consideration.

In the last column—or columns if more thanone restaurant is being compared—the sameevaluations and calculations are made for an ac-tual restaurant or restaurants. From the totals,merit ranges and merit indexes are established bydividing the totals by the sum of the criterionweights. Because in this case the scales wereconverted to a 1 to 10 range, ranges and meritindexes are also in the range of 1 to 10. Tobring the final values back into the star system,appropriate stars may be matched to ranges; thesample restaurant would be awarded 11⁄2 stars.

The basic criterion function model may befurther enhanced by introducing fuzzy mathemat-ics for uncertainty, range arithmetic, confidencevalues and other means for making and combin-ing judgments. Objective data (such as costs) canbe included with subjective data, and reversedscales where larger numbers represent lowervalue can be accommodated.

Criterion function modeling represents a sig-nificant improvement over the earlier presentedmethods because it allows criteria to be objecti-fied and weighted, and multiple competing sub-jects to be more fairly evaluated. The techniqueis powerful for making choices and even morepowerful as a means for understanding the im-pact of decisions. By playing "what if" gameswith the data in spreadsheet software, it is possi-

ble to gain insight about the choice-making pro-cess as well as the choices.

A Hierarchical, Process-oriented Model

In spite of its great flexibility and ability to in-corporate a broad range of criteria, the criterionfunction model has some built-in limitations thatdiminish its usefulness for evaluating systems,services and other complex man-made entities.This is particularly a problem for planners, de-signers and managers interested in improvingperformance or making comparisons among sys-tems for effectiveness of operation.

First, the conventional approach does notvery well suit the purposes of planning and de-sign. The focus of evaluation in the criterionfunction model and similar models is the qualityof the delivered experience, rather than the quali-ty of the process that delivers it. Criteria in thesemodels are almost always derived from the char-acteristics desired in the end result: in the restau-rant example, good food, good service, pleasantambiance, etc. From these, component categoriesmay be extrapolated (food quality, presentation,attentiveness, knowledgeability, interior design,lighting), but all still address the end result, onlytenuously connecting quality with the processthat creates it. When food quality is a criterionfor the evaluation of a restaurant, judgment isrendered through a complex assessment of tastesand preparations, all concerned with the product,with negligible consideration—if any—for howthe food quality was produced. For the systemplanner and designer, the process and the systemthat implements it are the primary subjects of in-terest.

Second, criteria are not associated with spe-cific, differentiated functions of the system, mak-ing it difficult to establish exactly where thedesign of the system is failing or doing well. Inthe food quality example, the food that reachesthe table for evaluation is the result of the inter-action of many human and system functions (di-rected actions that the system or its usersperform) that execute the restaurant’s particularapproach to the creation of cuisine. Failure ofany one or combination of these may be thecause of poor quality; and the special success ofone or more may be the source of unusually highquality. Either way, failure to involve system

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


operations explicitly as criteria leaves quality un-associated with the means of production.

Third, the system as the object of evaluationis not expressed in sufficient detail to identifythe individual failures and successes of actualsystem components. Evaluating a "restaurant" asa total concept against a series of criteria re-quires mentally calling up a different set of therestaurant’s properties for each criterion. Thoseproperties are presumed to be known to the eva-luator even though they have not been explicitlydescribed! This probably accounts for much ofthe disagreement that arises among multiple eva-luators, who inevitably conjure up different prop-erty composites, viewing the subject as theproverbial blind men "see" the elephant.

The Evaluation ToolTo build a tool for evaluation well suited to theneeds of the planner and designer as creators ofa quality-producing system—as well as the man-ager as maintainer of quality production—boththe nature of the criteria and the characterizationof the system being evaluated need to bechanged, and the evaluation process itself mustbe modified to take advantage of the new struc-tures.

Replacing the Criteria. Replacing the criter-ia, is a "structure" that abstracts the system as anorganization of requirements that the systemmust meet to perform well. This structure actslike a set of criteria in that components of thesystem being evaluated are scored against eachof its elemental requirements for how well theyperform. It can be constructed in two differentforms that evaluate the system at two differentlevels of abstraction.

At the higher degree of abstraction, the struc-ture created is a Policy Structure grounded in aseries of Defining Statements at its lowest levelthat are the positions on issues of policy that thesystem should exemplify. Higher levels in the hi-erarchy are successive layers of policy categoriesdecreasing progressively in number to a PolicyMerit assessment at the peak. For a thoroughdiscussion of Defining Statements and projectdefinition at the policy level generally, seeOwen, 2001a and 2005. For the purposes of thispaper, only one criteria structure need be pre-sented and discussed and, because it is more fre-quently used, that one will be the FunctionStructure.

A Function Structure is at a level of abstractionclosely matching that of the system to be evalu-ated. Rather than high level policy requirementsprescribed in a Policy Structure, it designates ac-tions to be taken to fulfill the objectives of thesystem’s operation. Functions express what thesystem or its users must do. Because it is hierar-chical, like the Policy Structure, the FunctionStructure also allows functionality to be consid-ered at different levels of categorization and de-tail. It is an operational model of the system as aprocess—in particular, the process that shouldexist.

Construction of a Function Structure proceedstop down through three levels of description,each different in kind and appropriate in charac-ter to the level of action involved. The three le-vels, from the top, are: Modes of Behavior,Activities and Functions (Figure 2).

Three qualitatively different kinds of levels:

A. Mode: Major condition of operation or behavior B. Activity: Purposeful performance C. Function: System or user action

System

Mode Mode

Submode Submode

Activity ActivityActivity

FunctionFunctionFunctionFunction

.

.

.Function

FunctionFunction

.

.

.Function

FunctionFunctionFunction

.

.

.Function

A.

B.

C.

Figure 2 The Function Structure

Modes. Modes are the major forms of behaviorthat the system exhibits or works within. The an-alyst preparing a Function Structure begins atthis level, considering the qualitatively differentways in which the system operates. There areseldom more than 10 to 15 modes. To distin-guish them from other action entities in theFunction Structure, they are named to suggest"classes" of operation. The suffix -tion frequentlyis used to specify a Mode (e.g., Operation as op-posed to Operating), but other noun forms workequally well as long as they carry with them asense of action. Figure 3 shows a simplifiedIn

stitu

te o

f Des

ign,

Illin

ois

Inst

itute

of T

echn

olog

y htt

p://w

ww.id

.iit.ed

u


302 System Support

103StagingMeals

102Producing

Meals

101Menu

Planning

201Food and Drink

Preparation

104Sanitizing

203 Space

Optimization

205Change

Management

RestaurantFunction Structure

105Preparing

for Service

107ProvidingService

106Accommo-

datingCustomers

108Creating

Ambiance

109Controlling

SpaceArrangement

110AcquiringMaterials

111Managing

Staff

112Complyingwith Lawsand Codes

113Maintaining

Environment114

Marketing115

Adapting

202Service

204Routine Conditions

Management

301Operation

8.Prepare foroperation

9.Prepareingredients

10.Preparedrinks

11.Communi-cate orders

12.Cook food

13.Coordinatetiming ofdishes

14.Control quality andconsistency

15.Finish/ platemeals

16.Assembleorder

17.Completepresentation

29.Respondto inquiries

30.Acceptreserva-tions

31.Supportparking

32.Greet customers

33.Store customerbelongings

34.Host wait-ing guests

35.Coordinateseating

36.Escort totable

37.Engagecustomers

38.Take foodand drinkorders

39.Communi-cate withstaff

40.Bring drinks

41.Providecomple-mentaryservices

42.Serve meal

43.Ensure satisfaction

44.Arrangepayment

45.Packageleftovers

46.Support/reinforcetheme

47.Tune decorto supportevents

48.Adjustsound quality

49.Control airquality

50.Control temperature

51.Optimizelighting

52.Perform signature rit-uals

53.Optimizestaff traffic

54.Facilitateguest traffic

55.Indicatedirectionsfor guests

56.Modify spatiallayout forevents

57.Respond toweatherconditions

58.Purchasefood/ drinksupplies

59.Replaceexpendablesupplies

60.Receiveshipments

61.Store per-ishables

62.Store non-perishables

63.Storeexpendablesupplies

64.Recruitemployees

65.Train staff

66.Monitorcareerdevelop-ment

67.Schedulework plan

68.Coordinatestaff work

69.Evaluateperform-ance

70.Motivateemployees

71.Comply withfinancialregulations

72.Comply withhealth code

73.Comply withfire/ buildingcode

74.Comply withemploymentcode

81.Executemarketingplan

82.Executecommunica-tions plan

83.Developpromotions

84.Build publicrelations

85.Advertise

86.Reinforceidentity

87.Promoteevents

88. Solicitcustomerfeedback

89.Monitordiningtrends

90.Respond tofeedback &reviews

91.Analyzecompetitors’offerings

92.Assess &redefineofferings

93.Adjustambiance

94.Confirmstrategy

1. Define mealmenu (B, L, D)

2.Defineoptions &combina-tions

3.Define drinkmenu

4.Suggestdishes

5.Prototype/test dishes

6.Evaluateresponse

7.Make sea-sonal adjust-ments

18.Gather/ sortutensils &tableware

19.Clean utensils &tableware

20.Store uten-sils & table-ware

21.Clean surfaces

22.Control foodquality

23.Preparewaiter stations

24.Stock bar

25.Preparecomple-mentaryservices

26.Set tables

27.Clear tables

28.Conductshift meeting

75.Managewasteremoval

76.Clean workspace

77.Clean linens

78.Maintainequipment

79.Maintaininfrastruc-ture

80.Repair fur-nishings &environment

Figure 3 Sample Function Structure for a Restaurant

Function Structure for a restaurant. In it, theModes are just Operation and System Support.Under them are Submodes, still major forms ofbehavior, but at a more specific level of descrip-tion. For the restaurant example, these are Foodand Drink Preparation, Service, Space Optimiza-tion, Routine Conditions Management andChange Management.Activities. Below the high-level Modes and Sub-modes, are Activities. At the Activity level, thenaming convention changes to fit a different con-ception of action, one that brings the analysis toa more dynamic level and closer to specific ac-tion descriptions. Activities are "purposeful per-formances". Much like the scenes of a play, theyidentify discrete collections of actions that worktoward a specific goal.

Activities are named using the gerund formof verbs—verbs with -ing endings that turn verbforms into noun forms expressing action withduration. Operating is an Activity; Operation isa Mode. For the Food and Drink Preparationsubmode, the Activities are: Menu Planning,Producing Meals, Staging Meals and Sanitizing.

Functions. Functions are the fundamental el-ements of the Function Structure. At this level,actions are described specifically enough thatthey can be used as criteria for evaluating the

system. A good system should be able to per-form all of the prescribed Functions well.

The issue of "completeness" surfaces normal-ly here. Does a hierarchically developed set ofFunctions completely and uniquely describe thesystem? The answer most certainly is "no". Forany set of Functions, it is always possible to findone more, if not by simple addition, by subdivid-ing a more general Function into two more spe-cific ones. A better question is, "Does ahierarchically developed set of Functions do amore thorough job of covering the descriptionthan an otherwise developed list? I think that theanswer to that is "yes". The measures of designare relativistic. If a design or plan (or in thiscase, the basis for a plan) is "better" than whatwe have been able to do previously, it is a suc-cess.

As in the construction of other levels, the se-lection and specification of Functions shouldseek to maintain balance and coverage withineach Activity. For form, the Function takes averb phrase, the form best able to capture neces-sary detail while conveying the sense of actioninherent in a system. For the Activity ProducingMeals in the example, the Functions are: Preparefor operation, Prepare ingredients, Preparedrinks, Communicate orders, Cook food, Coordi-

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


203StagingMeals

202Producing

Meals

201Menu

Planning

301Food and Drink

Preparation

204Sanitizing

303Space

Optimization

Restaurant

205Preparing

for Service

207ProvidingService

206Accommo-

datingCustomers

208Creating

Ambiance

209Controlling

SpaceArrangement

302Service

401Operation

102Develop-

mentProcedure

103Ingredient

Storage106

Prepara- tion Protocol

107Plating/

Finishing Area

108Service

Circulation

109Kitchen

Communi- cations Protocol

110Cleaning Equipment

111Sanitation Procedure

112Service Supply

114Inquiry

Response system

115Parking Provision

101ConceptSources

104Preparation

Zones

105Specialty Equipment

113Set/Clear Procedure

116Hosting/ Waiting

Area

118Service Support

119Staff Com-munication Procedure

122SoundControl

123Impression-

creating Décor

124Air Quality

Control

125Floor Plan

126Adaptive

Furnishings

117Seating Protocol

120Customer

Communica-tion Policy

121Mood

Lighting

402 System Support

305Change

Management

Restaurant


211Managing

Staff


213Maintaining

Environment214

Marketing215

Adapting


Management

127Vendor

Relation-ships

128Storage

130TrainingProgram

131Development

andAssessment

Program

129InventoryProtocol

132Manage-

mentPolicy

133Safety

Provisions134

ComplianceProcedures

135Waste

DisposalSystem

136CleaningProtocol

137Mainte-nance

Program

139IdentityRein-

forcement140

AnalysisTools

138Marketing/Communi-

cations

141Response

Mechanism

System Structure

Figure 4 Sample System Structure Archetype for a Restaurant

nate timing of dishes, and Control quality andconsistency.

Replacing the Object of Evaluation. In con-ventional evaluation processes, the object ofevaluation is normally a whole entity. As criteriaare addressed, characteristics of importance aresummoned and evaluated, criterion by criterion.In the new process, the object of evaluation, likethe criteria, is represented hierarchically as astructure—a System Structure (Figure 4). Thismodel submits multiple elements of the systemto the scrutiny of multiple criteria and enablesboth system and criteria to be organized andstudied at different levels of detail.

The System Structure is constructed from theupper levels of the Function Structure redirectedto categories of operational components at lowerlevels of the hierarchy. These categories collectclasses of "actuation" and, ultimately, the actualcomponents that outfit the system. The result is asystem archetype or template at a class or cate-gory level which can be fitted out with actuallowest-level System Elements for a specific sys-tem (or systems, as they remain within the rangedefined by the archetype). In the restaurant ex-ample, this enables an analyst to compare restau-rants within the same approximate competitiverange (Figure 5); in this case, covering mid- toupper-mid-range restaurants—not as high as the

peak level of fine dining, but well above thelevel of franchised chain restaurants.

System Structure Levels. A System Struc-ture is essentially a hierarchy of decreasing le-vels of abstraction that organizes the componentsof a system. When developed as a product of theStructured Planning process, the structure is anatural byproduct of the synthesis of the systemand its System Elements. When developed inde-pendently for the purpose of evaluation, it can becreated most easily from the upper levels of theFunction Structure (which will already have beenprepared in earlier steps of the evaluation pro-cess).

Mode and Submode levels from the FunctionStructure can be used directly. Depending on thelevel of coverage achieved at this point, the Ac-tivity level may also be carried over (as wasdone in the restaurant example of Figure 4), orthe style of description may change over at thispoint toward component depiction. At the high-est categorization level, categories answer thequestion, "What fundamental functional categor-ies are necessary to support this Submode or Ac-tivity?" At lower levels of categorization,categories move from general class descriptionsto more specific property descriptions until alevel is reached at which the next level wouldrequire the instantiation of an actual element of

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


203StagingMeals

202Producing

Meals

201Menu

Planning

301Food and Drink

Preparation

204Sanitizing

303Space

Optimization

Restaurant

205Preparing

for Service

207ProvidingService

206Accommo-

datingCustomers

208Creating

Ambiance

209Controlling

SpaceArrangement

302Service

401Operations

102Develop-

mentProcedure

103Ingredient

Storage106


107Plating/

Finishing Area

108Service

Circulation

109Kitchen


110Cleaning Equipment


112Service Supply

114Inquiry

Response system

115Parking Provision

101ConceptSources

104Preparation

Zones


113Set/Clear Procedure

116Hosting/ Waiting

Area

118Service Support

119Staff Com-munication Procedure

122SoundControl

123Impression-

creating Décor

124Air Quality

Control

125Floor Plan

126Adaptive

Furnishings

117Seating Protocol

120Customer

Communica-tion Policy

121Mood

Lighting

402 System Support

305Change

Management


211Managing

Staff


213Maintaining

Environment214

Marketing215

Adapting


Management

127Vendor

Relation-ships

128Storage

130TrainingProgram

131Development

andAssessment

Program

129InventoryProtocol

132Manage-

mentPolicy

133Safety

Provisions134

ComplianceProcedures

135Waste

DisposalSystem

136CleaningProtocol

137Mainte-nance

Program

139IdentityRein-

forcement140

AnalysisTools

138Marketing/Communi-

cations

141Response

Mechanism

203StagingMeals

202Producing

Meals

201Menu

Planning

301Food and Drink

Preparation

204Sanitizing

102Develop-

mentProcedure

103Ingredient

Storage106


107Plating/

Finishing Area

108Service

Circulation

109Kitchen


110Cleaning

Equipment


101ConceptSources

104Preparation

Zones


1.Research Travel2.Recipe Books

3.Dish Tastings4.IngredientsProcurement

5.Preparation& Walk-inCoolers 6.Dry Storagein Kitchen

7.IngredientsPrep

8.Blast Chiller9.Wood BurningOven

10.Food RecipeClipboards11.Kitchen ProcessSys Clipboards

12.Plating Zone

13.Expeditors14.POS OrderPrinters

15.EmployeeHierarchy16.BilingualCommunications

17.Heated Cleaningfor Glassware18.DesignatedPot CleaningArea

19.MandatoryCleaningProcess

203StagingMeals

202Producing

Meals

201Menu

Planning

301Food and Drink

Preparation

204Sanitizing

102Develop-

mentProcedure

103Ingredient

Storage106


107Plating/

Finishing Area

108Service

Circulation

109Kitchen


110Cleaning

Equipment


101ConceptSources

104Preparation

Zones


22.Visit OtherRestaurants 23.Trade Journals

24.Partnerships

25.Committee andRating Sheet26.As a SpecialFirst

27.LabeledContainers

28.Food Zones29.Cook Zones

20.Buffalo Chopper30.Wok41.Grill Griddle

17.Drahma18.Individual Prep Sheet19.Build42.Line Check

43.Heat Lamps

16.2-Door Flow

3.Allergy Avoidance44.2-Ticket System45.CarryoutOrder Sheet

31.Conveyer CleaningMachine32.Dirty DishShelving Placement

33.ScrapeDump RackProcess34. Dry StoreShelvingProcess

Restaurant: Osteria Via Stato

Restaurant: Ben Pao

Figure 5 Comparison of Two Actual Restaurants Using the Same System Structure Archetype

the system under study. At this point, the SystemStructure is complete as an archetype usable forany system within the comparison group.

The Evaluation Process

The evaluation process is one of scoring andvalue accumulation through calculation. SystemIn

stitu

te o

f Des

ign,

Illin

ois

Inst

itute

of T

echn

olog

y htt

p://w

ww.id

.iit.ed

u

Janu

ary,

2007


Evaluation MatrixScored on a 5-place scale

SysEl

Submode

Mode

System

Submode Submode . . SubmodeSubmode

SysEl SysEl SysEl SysEl SysEl SysEl SysEl SysEl SysEl SysEl . . . . . . SysEl

Mode . . . . . . . . . . . Mode

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Activity

Sub-mode

Mode

Activity

Activity

System

ActivityFunction

Sub-mode

Activity

Sub-mode

Mode

FunctionStructure

SystemStructure

Strongly Supports: System Element is an excellent solution or provides additional strongsupport toward fulfilling the FunctionFirmly Supports: System Element is a good solution or provides additional good supporttoward fulfilling the Function

No Effect: System Element has no bearing on the FunctionObstructs: System Element is not too good a solution or mildly interferes with fulfilling theFunctionFirmly Obstructs: System Element is a poor solution or actually obstructs fulfilling theFunction

+3

+2

0–1

–2

7-Place Evaluation Scale

Supports: System Element is an adequate solution or provides some support toward fulfilling the Function

+1

Strongly Obstructs: System Element is a bad solution or seriously obstructs fulfilling theFunction

–3

Strongly Supports: System Element is an excellent solution or provides additional strongsupport toward fulfilling the FunctionSupports: System Element is a good solution or provides additional support toward fulfilling the FunctionNo Effect: System Element has no bearing on the Function Obstructs: System Element is not too good a solution or obstructs fulfilling the FunctionStrongly Obstructs: System Element is a bad solution or seriously obstructs fulfilling theFunction

+2

+1

0–1

–2

5-Place Evaluation Scale

Figure 6 5-place and 7-place Scales for Scoring Relationships

Elements are scored for how well they supportthe Functions. Because in some cases they mayactually obstruct or impede fulfillment of Func-tions unintentionally, provision is made for bothnegative and positive scoring.

The Scoring ScaleScoring is qualitative, conducted on a bidirec-tional, five- or seven-place scale (Figure 6). Atthe central point of this scale, a rating is neutralindicating that the System Element has no effecton the Function, neither supporting its fulfillmentnor obstructing it. Positions on the positive side

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


-3-2-10

+1+2+3

quantitativequ

alita

tive

-3-2-10

+1+2+3

quantitative

qual

itativ

e

-3-2-10

+1+2+3

quantitative

qual

itativ

e

-3-2-10

+1+2+3

quantitative

qual

itativ

e

-3-2-10

+1+2+3

quantitativequ

alita

tive

-3-2-10

+1+2+3

quantitative

qual

itativ

e

-3-2-10

+1+2+3

quantitative

qual

itativ

e

-3-2-10

+1+2+3

quantitative

qual

itativ

e

Figure 7 Mapping Quantitative Data to a 7-place Qualitative Scale

of the scale allow increasing degrees of support;positions on the negative side register increasingdegrees of obstruction.

Although it requires additional processing,quantitative information, where it exists, may beincorporated in evaluations by mapping values tothe qualitative scale. Figure 7 illustrates the pro-cess diagrammatically for a variety of classic re-lationships. The range of possible quantitativevalues for a System Element’s performance rela-tive to a Function is plotted against the 5- or7-place scale being used for qualitative judg-ments, and the resulting plot is used to translatescores for the given System Element.

Although component elements of any systemare designed to meet the needs of specific func-tions, it is not unusual that they have effects onother functions. The more complex the system,the more likely it is that there will be collateraleffects of System Elements on Functions otherthan those for which they were intended. Theremay even be surprises, some of them undesira-ble. In any case, it is important to examine thefull range of possible relationships and to recog-nize active associations, positive or negative, sothat they may be directly addressed. Thestructure-against-structure formulation of theevaluative process guarantees this.

Graphic data collection forms of variouskinds can be used to simplify the task of scoring.The form in Figure 8 was created for use in theexample restaurant evaluation. The master formshowing all Functions was multiply replicatedwith the title for each System Element added byhand. Focusing on the System Element allowed

the analysis team to consider the System Ele-ment vs all Functions (rather than Function vs allSystem Elements) as the easier of the two waysto approach the scoring.

Accumulating Values for Functions and System Elements

Vfinal = Vpos + Vneg

Vpos =1

[(v+max+ 2) – v+

2 ]2

1[(v+

max+ 2) – v+3 ]3

+ 1[(v+

max+ 2) – v+npos

]npos+ . . .

Vneg =1

[(v+max+ 2) + v–

2 ]2

1[(v+

max+ 2) + v–3 ]3

– 1[(v+

max+ 2) + v–nneg

]nneg– . . .

v+1 +

v–1 –

(1)

(2)

(3)

+

–

Values for Functions and System Elements arecalculated similarly using an algorithm and equa-tions that force accumulating values to be as-ymptotic to limits readily interpretable in termsof the scoring scale. Steps in the algorithm are:

1. If the rating scale does not have 0 for itsneutral value, transform the scale and convert allvalues. For data entry purposes, it is convenientto use unsigned scales such as 1 - 5 or 1 - 7,

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


103

Stag

ing

Mea

ls

102

Prod

ucin

gM

eals

101

Men

uPl

anni

ng

201

Food

and

Drin

kPr

epar

atio

n

104

Sani

tizin

g

203

Spac

eO

ptim

izat

ion

Res

taur

ant

Eval

uatio

n Pr

ojec

t7-

Pla

ce S

cale

105

Prep

arin

gfo

r Se

rvic

e

107

Prov

idin

gSe

rvic

e

106

Acc

omm

o-da

ting

Cus

tom

ers

108

Cre

atin

gA

mbi

ance

109

Con

trol

ling

Spac

eA

rran

gem

ent

202

Serv

ice

301

Ope

ratio

n

302

Syst

em S

uppo

rt

205

Cha

nge

Man

agem

ent

110

Acq

uirin

gM

ater

ials

111

Man

agin

gSt

aff

112

Com

plyi

ngw

ith L

aws

and

Cod

es

113

Mai

ntai

ning

Envi

ronm

ent

114

Mar

ketin

g11

5A

dapt

ing

204

Rou

tine

Con

ditio

nsM

anag

emen

t

57. Respond to weather conditions

56. Modify spatial layout for events

55. Indicate directions for guests

54. Facilitate guest traffic

53. Optimize staff traffic

52. Perform signature rituals

51. Optimize lighting

50. Control temperature

49. Control air quality

48. Adjust sound quality

47. Tune décor to support events

46. Support/reinforce theme

45. Package leftovers

44. Arrange payment

43. Ensure satisfaction

42. Serve meal

41. Provide complementary services

40. Bring drinks

39. Communicate with staff

38. Take food and drink orders

37. Engage customers

36. Escort to table

35. Coordinate seating

34. Host waiting guests

33. Store customer belongings

32. Greet customers

31. Support parking

30. Accept reservations

29. Respond to inquiries

28. Conduct shift meeting

27. Clear tables

26. Set tables

25. Prepare complementary services

24. Stock bar

23. Prepare waiter stations

22. Control food quality

21. Clean surfaces

20. Store utensils & tableware

19. Clean utensils & tableware

18. Gather/sort utensils & tableware

17. Complete presentation

16. Assemble order

15. Finish/plate meals

14. Control quality and consistency

13. Coordinate timing of dishes

12. Cook food

11. Communicate orders

10. Prepare drinks

9. Prepare ingredients

8. Prepare for operation

7.Make seasonal adjustments

6. Evaluate response

5. Prototype/test dishes

4. Suggest dishes

3. Define drink menu

2. Define options & combinations

1. Define meal menu (B.L.D.)

58. Purchase food/drink supplies

59. Replace expendable supplies

60. Receive shipments 61. Store perishables

62. Store non-perishables

63. Store expendable supplies

64. Recruit employees 65. Train staff

66. Monitor career development

67. Schedule work plan

68. Coordinate staff work

69. Evaluate performance

70. Motivate employees

71. Comply with financial regulations

72. Comply with health code

73. Comply with fire/building code

74. Comply with employment code

75. Manage waste removal 76. Clean work space77. Clean linens

78. Maintain equipment

79. Maintain infrastructure

80. Repair furnishings & environment81. Execute marketing plan

82. Execute communications plan

83. Develop promotions

84. Build public relations 85. Advertise 86. Reinforce identity87. Promote events

88. Solicit customer feedback

89. Monitor dining trends

90. Respond to feedback & reviews

91. Analyze competitors’ offerings

92. Assess & redefine offerings

93. Adjust ambiance94. Confirm strategy S

yste

m E

lem

ent

Cat

egor

y

Syst

em E

lem

ent

Wei

ght

(Impo

rtanc

e to

Act

ivity

)

Sys

tem

Ele

men

tA

ctiv

ity

Stro

ngly

Sup

port

s: S

yste

m E

lem

ent i

s an

exc

elle

nt s

olut

ion

orpr

ovid

es a

dditi

onal

stro

ng s

uppo

rt to

war

d fu

lfilli

ng th

e Fu

nctio

nFi

rmly

Sup

port

s: S

yste

m E

lem

ent i

s a

good

sol

utio

n or

pro

vide

sad

ditio

nal g

ood

supp

ort t

owar

d fu

lfilli

ng th

e Fu

nctio

n

No

Effe

ct: S

yste

m E

lem

ent h

as n

o be

arin

g on

the

Func

tion

Obs

truc

ts: S

yste

m E

lem

ent i

s no

t too

goo

d a

solu

tion

or m

ildly

inte

rfere

s w

ith fu

lfilli

ng th

e Fu

nctio

nFi

rmly

Obs

truc

ts:

Sys

tem

Ele

men

t is

a po

or s

olut

ion

or a

ctua

llyob

stru

cts

fulfi

lling

the

Func

tion

7 6 4 3 2Eval

uatio

n Sc

ale

Eval

uatio

n Fo

rm

No.

Nam

e

16

1116

2126

3136

4146

5156

5661

6671

7681

8691

Supp

orts

: Sys

tem

Ele

men

t is

an a

dequ

ate

solu

tion

or p

rovi

des

som

e su

ppor

t tow

ard

fulfi

lling

the

Func

tion

5

Stro

ngly

Obs

truc

ts: S

yste

m E

lem

ent i

s a

bad

solu

tion

or

serio

usly

obs

truct

s fu

lfilli

ng th

e Fu

nctio

n1

58

Figure 8 Master Form for Scoring Function Data for a System ElementInst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


Evaluation MatrixSample calculations

SysEl

Submode

Mode

System

Submode Submode . . SubmodeSubmode

SysEl SysEl SysEl SysEl SysEl SysEl SysEl SysEl SysEl SysEl . . . . . . SysEl

Mode . . . . . . . . . . . Mode

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Function

Activity

Sub-mode

Mode

Activity

Activity

System

ActivityFunction

Sub-mode

Activity

Sub-mode

Mode

FunctionStructure

SystemStructure

Vpos = 2 + 1[(2 + 2) – 2]2

1[(2 + 2) – 1]3

1[(2 + 2) – 1]4

+ +

= 2.299 + 0.000 = 2.299

Vneg = 0.000

Vfinal

= 2.299

Vpos = 2 + 1[(2 + 2) – 1]2

1[(2 + 2) – 1]3

+

= 2.148 + (–1.000) = 1.148

Vneg = –1.000

Vfinal

= 2.148Function:

SystemElement:

Figure 9 Sample Calculations for the Value of a Function and a System Element

where the neutral point is 3 or 4. Conversion ishandled easily within the calculation section ofthe computer program. The maximum scalevalue is V+

max.2. Sort positive ratings, v+, highest first; sort

negative ratings, v–, lowest first.3. Calculate composite values for the positive

series and negative series using equations (1) and(2).

4. Combine the composite values for the finalvalue using equation (3).

For the positive calculation, the diminishingseries limits the value to a value .5 above themaximum value given during scoring (+1 or +2

for a 5-place scale; +1, +2 or +3 for a 7-placescale). For the negative calculation, the seriescalculation limits the value to .5 below the mini-mum value given during scoring (–2 or –1 for a5-place scale; –3, –2 or –1 for a 7-place scale).Values are calculated for each Function using thescores given it for each System Element. Valuesare also calculated for all System Elements usingthe scores given them for each Function. Be-cause of the asymptotic behavior of the seriescalculation, a Function will have for its overallpositive score a value no lower than its singlehighest score, and no higher than that value plus.5. Similarly, the negative portion of its score

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


Function 1 3 1.111

Function 2 2 1.148

Function 3 5 1.165

Activity 1 4

[(3)(1.111) + (2)(1.148) + (5)(1.165)]

(3 + 2 + 5)= = 1.145

w vvv w w

1.145

vw

V =! wivi

i = 1,n

! wii = 1,n

Activity 1V

vw

Figure 10 Calculation of Nodal Values Using Weights

will always be at least as bad as its single lowestscore, and no worse than that value minus .5.This supports a policy that holds that a Functionor System Element is served as well as the bestand worst evaluations given for it. Subsequentpositive or negative evaluations can add to orsubtract from that in the series calculation, butnot substantially enough to raise or lower the ul-timate value to the next scale level. Then, whenrated on a 7-place scale, a Function that receivesa single –3 score can never achieve an overallrating of +1 no matter how high or how manypositive scores it receives: similarly, a SystemElement that receives a single +3 score cannever drop overall to the –1 level no matter howlow or how many negative scores it receives.

If the values were simply averaged instead ofusing the series calculation, the contribution of asingle strong system Element to fulfilling aFunction could be quickly lost. For example, theset of scores {+3, +1, +1, +1} produces an aver-age value of 1.5, where the series calculationproduces 3.08. In the averaged calculation, theFunction appears not to be well fulfilled, where-as in the series calculation, the single +3 scoreguarantees an evaluation of "strong fulfillment".Similarly, averaging {–3, –1, –1, –1} results in–1.5, quite different from the –3.08 calculatedwith the series equation. The series calculation isbased on the concept that a function of a systemneeds only to be fulfilled once; multiple contri-butions to fulfillment are welcome supplementsfor redundancy and reliability, but proper evalua-tion should recognize and communicate thestrength of the highest score.

When positive and negative results aremerged, however, cancellation unavoidably ob-scures some of the transparency built into the se-

ries calculation. Fortunately, in practice, negativescores are usually scarce, far fewer than positivein almost any evaluation; but the situation doesarise. To prevent inadvertent misinterpretation ofevaluation scores, a "marking" system is builtinto the graphic presentation of final results. Itwill be introduced and discussed in the Commu-nication section of this paper.

Using the data from the simplified exampleof Figure 6, the values for a sample Functionand System Element are calculated in Figure 9:1.148 and 2.299 respectively. The calculationsare shown in Figure 9.

Hierarchical CalculationsBecause of the hierarchical formulation of bothevaluative criteria and the system under evalua-tion, it is possible to measure performance atnodal positions throughout both structures. Thisenables the analyst to view the system at varyinglevels of detail from the elemental to the all in-clusive. At the apex, a single value integrates allvalues below into a Functional Merit Index, aPolicy Merit Index or a System Merit Index.

Weights. Once constructed, a Function Struc-ture and a System Structure archetype can beused to compare systems. In most cases, the onlyadjustments necessary are the instantiations inthe archetype of System Elements specific to thesystems being compared. If there are more fun-damental variations between systems—usuallycaused by differences in the contexts in whichthe systems operate—weights can be employedto emphasize those aspects of functionality mostrelevant to the individual contexts.

Weights may be used at all structural levelsto reflect relative importance. Various strategiesmay be employed for determining weights, butusually these will reflect the relative importance

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


Function 1 3 0.811

Function 2 2 1.148

Function 3 5 1.165

Activity 1 4 1.055

vw

0

1

0

0

!critical

V =" wivi

i = 1,n

" wii = 1,n

!i = 1 (critical)

unless, for any i, i = 1, n

vi < 0.0 (unacceptable)and

then:

for n son nodes of a parent node

Function 1 3 0.811

Function 2 2 1.148

Function 3 5 1.165

Activity 1 4 1.055

vw0

0

0

0

!non-critical

Function 1 3 0.811

Function 2 2 1.148

Function 3 5 1.165

Activity 1 4 –2.5

vw

0

2

0

0

!

hypercritical

Function 1 3 –0.011

Function 2 2 1.148

Function 3 5 1.165

Activity 1 4 –2.5

vw

0

1

0

0

!critical

!i = 2 (hypercritical)

vi < 1.0 (unsatisfactory)and

or

Case 1

Case 2

Case 3

Case 4

V = vmin – .5

vw!

vw!

vw!

vw!where vmin is thelowest scale value

Figure 11 The Effect of Criticality on Values

to each other of elements within a category,given the contextual circumstances.

Figure 10 diagrams the inclusion relationshipbetween an Activity and its three member Func-tions. Each of the Functions has a value, v, cal-culated from the scores given it for SystemElements. Each also has a weight, w, establish-ing its importance relative to the others in per-forming the Activity. The value for the Activityis calculated as a simple weighted average of thevalues for the Functions. The actual calculationfor this example is also shown in Figure 10.

Critical Values. In many systems, certaincomponent Functions or elements are critical.Their failure would be more serious than single-component failure and might cause failure oftheir parent processes, or even the entire system.To accommodate that possibility, the evaluationprocess has criticality factors, !, that can be as-signed to any element of either the criteria orsystem hierarchies.

Criticality is assigned at three levels. At level0 (the normal case) performance of the designat-ed element is not critical, and its value will beaveraged with other values no matter what itsscore. At level 1, performance is considered

critical, and a value below 0.0 will cause thevalue of the parent process or category to go tominimum calculable value (–2.5 or –3.5) withoutregard for sibling component values. At level 2,performance is considered hypercritical, and per-formance below 1.0 is cause for the value of theparent process or category to be driven to mini-mum calculable value. In all cases, satisfactoryperformance results in the value being normallyaveraged with other sibling values no matterwhat the criticality factor.

Figure 11 diagrams the calculation processfor an Activity with three Functions scored on a5-place scale. In Case 1, ! for Function 1 is setto 0 (off), and the three values for Functions 1, 2and 3 are averaged with their weights normallyusing the equation shown to establish the valuefor Activity 1 (1.055). In Case 2, ! is set to crit-ical (1) for Function 1, but the function’s perfor-mance is sufficiently high (> 0.0) that theactivity value is still calculated normally. InCase 3, ! is set to hypercritical (2), and thesame value (0.811) in the zone between unsatis-factory (< 1.0) and unacceptable (< 0.0) causesthe Activity value to go to –2.5, the lowest valuepossible on the 5-place scale. In Case 4, ! set to

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


critical (1) is sufficient to drive the activityvalue to –2.5 when Function 1’s value drops toan unacceptable level (< 0.0).

Communicating the Evaluation

The complexity of this evaluative process is fullyreflected in the tasks of calculation and commu-nication. The process is only practical when sup-ported by computer programs. A proprietarycomputer program, SYSEVAL, has been writtento handle calculations and produce graphic rendi-tions of the results in color. Output formats ofseveral kinds are selectable by content and form.

Content OptionsThree options are possible for content: a Func-tional Assessment, Policy Assessment and Sys-tem Assessment. A Functional Assessmentpresents results in the form of the original Func-tion Structure, showing values, weights and criti-cality factors for each Function and all higherlevel Activities, Submodes and Modes to an all-inclusive Functional Merit Index. For Policy As-sessments, the structure is similar with DefiningStatements at the bottom of the hierarchy andcategory descriptions continuing to a peak levelPolicy Merit Index. System Assessments haveSystem Elements at the lowest level with theircategories at one to three levels above and thenActivities (optionally) followed by Submodesand Modes to a top level System Merit Index.

Form OptionsPresentations are produced in color, following ei-ther a vertical or horizontal plan. Vertical hierar-chies branch from the Merit Index at the left tothe base level elements on the right arrangedvertically along the right side of the structure.The vertical format can be produced as a contin-uous hierarchy (for roll printing of a verticallyoriented, left-to-right structure) or paged for re-port generation on 81⁄2-inch or A4 width sheetsof paper. Horizontal hierarchies branch from thetop down so that base level elements are at thebottom of a roll-printed sheet scrolling from leftto right. For this format, a 24-inch (60 cm) paperwidth is used to allow more compact descriptionof hierarchies. The limitation imposed by main-taining a legible type size guarantees lengthyhorizontal or vertical "posters" for structures withmore than 100 Functions, Defining Statements or

System Elements. When large numbers of ele-ments are involved in an evaluation, the resultinghierarchies can extend to over three meters.

Figure 12 Evaluation Scale with Color and Line Thickness Assignments

Key

A. Excellent: (Strong support): from 3.0 to just less than 3.5 B. Very Good: (Firm support): from 2.0 to just less than 3.0

D. Unsatisfactory (No support or a consensus less than supporting): from 0.0 to just less than 1.0

E. Unacceptable (Obstruction): from just less than 0.0 to just greater than -2.0

G. Wholly (Strong obstruction): from -3.0 to -3.5 Unacceptable:

0.0 3.5-3.5 2.0-1.0 1.0-2.0

ABCDG

3.0-3.0

EF

C. Good: (Support): from 1.0 to just less than 2.0

F. Seriously (Firm Obstruction): from just less than -2.0 to just greater than -3.0Unacceptable

Graphic CharacteristicsImportant aids to interpretation are color and lineweight. Both are used to call attention to suc-cesses and failures within a hierarchy. Dependingon the evaluation scale used, an assignment ofone of five or seven colors is made to all blocksrepresenting elements and lines linking blocks.The color spectrum is related to the evaluationscales as shown in Figure 12. Greens and bluessignal successes; oranges and reds warn of prob-lems. Yellow denotes an unsatisfactory condi-tion, usually indicating conflict or uncertainty.The thickness of links is correlated to the valuesof the parent blocks they connect to offspring.Thicker lines trace success; thinner lines, prob-lems.

Each block in a hierarchy has identificationinformation: (1) type designation (Mode, Sub-mode, Activity, Function, etc.), (2) number, (3)title, (4) criticality factor, (5) criticality ring, (6)weight, (7) value, and (8) value disc. These com-ponents are depicted and described in Figure 13.

Two special block types are reserved for po-tentially ambiguous situations (Figure 14). Ared-flagged yellow block signals an entity forwhich there are no scores, positive or negative.If a Function or Defining Statement, that meansthat no System Elements support or obstruct itsfulfillment. If a System Element, it means that itneither serves nor obstructs any Function. Itsvalue will be 0.00, but that value will have beenobtained by the absence of any effective score.

The second special block type is a green-flagged yellow block. It signals the deliberatechoice of the analyst to take a category of the

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


System Structure out of the analysis, usually be-cause the analyst has decided that the category isnot relevant for a particular system for contextu-al reasons. By assigning a weight of 0 to the cat-egory, the analyst signals the program to ignoreit in all hierarchical calculations. The block isshown in the output structure with its 0 weightand 0.00 value, but is flagged green.

Two other graphic additions support interpre-tation of results. First, a lavender ring is super-imposed over a 1 or 2 (critical or hypercritical)rating shown for the criticality factor on anyblock. This helps the evaluator to note wherecriticality has been set and met or not met—particularly where it has been met, because therewill be a color change to red only when it hasnot been met.

The second graphic aid is a colored "valuedisc" printed beneath the value on any block inwhich a positive value has been diminished bynegative scores. By its color (following the colorscale), it indicates the most negative score re-ceived. As discussed earlier, the cancellation ofpositive and negative scores conceals thestrength of scores received; the value discs callattention to that. Blocks that have overall nega-tive values are already prominent by theirorange-red colors. Positive blocks, green andblue, and yellow blocks still in the positiverange, receive value discs to call attention to theexistence of negative scores.

Figures 15 and 16 show a sample FunctionalAssessment and System Assessment from therestaurant study in horizontal format. Structuresare created as pdf’s, intended for purposes oflegibility to be scaled and printed so that thewidth of blocks is approximately 1 inch (2.5cm).

Function 27Tune decor to support events

1 250 2.44

1. Type DesignationEntity represented: Function, Activity, Submode, Mode, System,Category, Defining Statement, System Element, etc.

2. NumberSequential numbers for Functions, Defining Statements andSystem Elements; level + number for higher level entities (e.g., 203 = level 2, entity 03)

3. TitleName or action

4. Criticality Factor 0 = non-critical; 1 = critical; 2 = hypercritical

5. Criticality RingAlert signal for critical and hypercritical blocks

6. WeightRelative importance of block to other blocks under same category

7. Value Weighted total score for block calculated from all scoresgiven for it

8. Value DiscColor-coded signal that negative scores have been given for theblock

123

45

6 78

Figure 13 Components of a Block

Figure 14 Special Blocks and Graphic Interpretation Aids

KeyFunction 27Tune decor to support events

1 250 2.44

Standard Block (Colors coded as above)Criticality Factor (0, 1 or 2) — ring signals "on"Weight relative to others in its classAccumulated Value (Scale above)

System Element 62Mosaic Window

0 001 0.00

System Category 112Event Planning

0 0 0.00

Red-Flagged Yellow Block• Entity unaffected by elements scored for it.• No scores given, either positive or negative.

Green-Flagged Yellow Block• System entity not evaluated for this specific structure.• Weight of 0 marks entity for isolation.• Value not included in higher-level calculations.

1 2 3

321

Defining Statement 112System should sup-port feeling of privacy

0 667 1.44

Value Discs• Positive total value includes negative score/s.

At leastone –1

At leastone –2

At leastone –3

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


Experimental System Evaluation Project

Osteria Via StatoFunctional Assessment

Figure 15 Sample Functional Assessment in Horizontal Format (Left half)

Activity 101Menu Planning

2 300 3.27

Function 1Define meal menu(B,L,D)

2 300 3.30

Function 2Define options &combinations

1 100 3.13

Function 3Define drink menu

1 200 3.47

Function 4Suggest dishes

0 100 3.15

Function 5Prototype/testdishes

1 200 3.44

Function 6Evaluate response

1 50 3.29

Function 7Make seasonaladjustments

0 50 2.08

Submode 201Food and DrinkPreparation

1 500 2.79

Activity 102Producing Meals

1 400 2.73

Function 8Prepare foroperation

1 100 3.39

Function 9Prepare ingredients

2 150 2.39

Function 10Prepare drinks

0 50 2.44

Function 11Communicate orders

1 100 2.39

Function 12Cook food

2 300 2.49

Function 13Coordinate timingof dishes

1 100 2.30

Function 14Control quality andconsistency

2 200 3.47

Activity 103Staging Meals

1 200 2.50

Function 15Finish/plate meals

1 600 2.16

Function 16Assemble order

1 100 2.16

Function 17Completepresentation

0 300 3.30

Activity 104Sanitizing

1 100 2.16

Function 18Gather/sortutensils & tableware

0 100 1.08

Function 19Clean utensils &tableware

1 350 2.15

Function 20Store utensils &tableware

0 100 1.07

Function 21Clean surfaces

1 150 2.10

Function 22Control food quality

1 200 3.31

Mode 301Operation

1 500 2.64

Activity 105Preparing forService

0 250 2.13

Function 23Prepare waiterstations

0 200 1.13

Function 24Stock bar

1 450 3.15

Function 25Prepare comple-mentary services

0 150 1.08

Function 26Set tables

0 200 2.13

Function 27Clear tables

1 200 1.08

Function 28Conduct shiftmeeting

1 100 3.13

Submode 202Service

1 400 2.50

Activity 106AccommodatingCustomers

1 250 1.85

Function 29Respond to inquiries

1 100 1.08

Function 30Accept reservations

0 100 3.08

Function 31Support parking

1 200 2.08

Function 32Greet customers

0 150 2.13

Function 33Store customerbelongings

1 100 1.08

Function 34Host waiting guests

0 100 1.02

Function 35Coordinate seating

0 200 2.16

Function 36Escort to table

0 50 1.02

Activity 107Providing Service

1 500 3.01

Function 37Engage customers

1 100 3.16

Function 38Take food and drinkorders

1 200 3.44

Function 39Communicate withstaff

1 100 3.38

Function 40Bring drinks

1 100 3.30

Function 41Provide comple-mentary services

0 100 2.08

Function 42Serve meal

1 100 3.27

Function 43Ensure satisfaction

0 200 3.44

Function 44Arrange payment

1 50 1.29

Function 45Package leftovers

0 50 1.08

Functional Merit401Osteria Via Stato

2.63

Submode 203Space Optimization

0 100 2.48

Activity 108Creating Ambiance

0 700 3.09

Function 46Support/reinforcetheme

0 200 3.50

Function 47Tune decor tosupport events

0 50 3.29

Function 48Adjust sound quality

0 100 2.08

Function 49Control air quality

1 100 3.08

Function 50Control temperature

1 100 3.08

Function 51Optimize lighting

0 100 3.08

Function 52Perform signaturerituals

0 150 3.15

Discussion

With the cooperation of the Chicago-based Let-tuce Entertain You restaurant organization, threemidrange fine dining restaurants were evaluatedby teams of three graduate students each fromthe Institute of Design. Testing was conducted tovalidate the evaluation model, explore potentialproblems associated with obtaining data andmaking judgments, and test the effectiveness ofthe hierarchical presentation scheme for commu-

nicating results. Management and staff in all res-taurants were highly supportive.

Recommendations from Restaurant TestsAs expected, it was difficult for the evaluationteams to acquire the expertise advisable for high-quality independent judgments. Despite prepara-tion with a range of good written materials onrestaurant design, planning and operation, andthe general knowledge available to all on the op-eration of restaurants, the teams quickly foundthat they needed the help of staff willing to

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


Figure 15A Sample Functional Assessment in Horizontal Format (Right half)

Activity 109Controlling SpaceArrangement

0 300 1.04

Function 53Optimize stafftraffic

1 250 0.01

Function 54Facilitate guesttraffic

0 250 -0.98

Function 55Indicationdirections for guests

1 200 3.27

Function 56Modify spatiallayout for events

0 100 2.08

Function 57Respond to weatherconditions

0 200 2.13

Activity 110Acquiring Materials

0 300 3.01

Function 58Purchase food/drinksupplies

1 300 3.49

Function 59Replace expendablesupplies

1 150 3.30

Function 60Receive shipments

1 200 3.29

Function 61Store perishables

1 150 2.39

Function 62Store nonperishables

1 100 2.23

Function 63Store expendablesupplies

1 100 2.30

Submode 204Routine ConditionsManagement

1 600 2.70

Activity 111Managing Staff

1 300 2.45

Function 64Recruit employees

1 200 2.08

Function 65Train staff

1 300 2.32

Function 66Monitor careerdevelopment

0 100 3.38

Function 67Schedule work plan

0 100 2.21

Function 68Coordinate staffwork

1 100 2.36

Function 69Evaluate performance

1 100 2.29

Function 70Motivate employees

1 100 3.15

Mode 302System Support

1 500 2.62

Activity 112Complying with Lawsand Codes

1 200 2.89

Function 71Comply withfinancial regulation

1 350 3.27

Function 72Comply with healthcode

1 250 2.39

Function 73Comply withfire/building code

1 200 2.44

Function 74Comply withemployment code

1 200 3.29

Activity 113MaintainingEnvironment

0 200 2.40

Function 75Manage wasteremoval

1 200 3.47

Function 76Clean work space

1 300 2.16

Function 77Clean linens

1 200 2.08

Function 78Maintain equipment

1 100 2.27

Function 79Maintaininfrastructure

1 100 3.08

Function 80Repair furnishings& environment

1 100 1.08

Submode 205Change Management

1 400 2.51

Activity 114Marketing

0 700 2.58

Function 81Execute marketingplan

0 200 1.08

Function 82Executecommunications plan

1 150 2.08

Function 83Develop promotions

0 50 3.27

Function 84Build publicrelations

0 100 3.38

Function 85Advertise

0 100 3.08

Function 86Reinforce identity

0 150 3.47

Function 87Promote events

0 50 2.08

Function 88Solicit customerfeedback

0 200 3.08

Activity 115Adapting

0 300 2.34

Function 89Monitor diningtrends

1 300 3.29

Function 90Respond to feedback& reviews

0 200 3.08

Function 91Analyzecompetitors’ offerings

0 150 1.08

Function 92Assess & redefineofferings

0 100 3.15

Function 93Adjust ambiance

0 100 1.00

Function 94Confirm strategy

1 150 1.08

speak candidly about operations. Conventionalobservation techniques were not easily employed,and the value of deep system knowledge wasquickly obvious. A clear recommendation wasthat reliable evaluations at this level of detail canbest be made either by content experts or knowl-edgeable management conscientiously self-evaluating their systems.

Recommendations directed toward data col-lection and interpretation included the following,all now implemented in the process (and dis-cussed above):

1. A 5-place scale was not granular enoughto rate relationships optimally. A 7-place scalewas added as an option.

2. A single level of criticality was not suffi-cient. Some Functions are critical, but less criti-cal than others. Two levels of criticality (criticaland hypercritical) have now been implemented.Because it is easy to miss seeing where criticali-ty factors are "on" when there are many blocksand the value of a block is high enough to avoidtriggering the criticality function, a contrasting

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007


Category 201Menu Planning

0 1 3.44

Category 101Concept Sources

0 1 3.47

SystemElement 1Research Travel

2 500 3.49

SystemElement 2Recipe Books

1 500 3.44

Category 102DevelopmentProcedure

0 1 3.42

SystemElement 3Dish Tastings

1 600 3.50

SystemElement 4IngredientsProcurement

1 400 3.31

Category 103Ingredient Storage

0 1 1.72

SystemElement 5Preparation &Walk-In Coolers

1 600 2.16

SystemElement 6Dry Storage InKitchen

0 400 1.07

Category 202Producing Meals

0 1 1.68

Category 104Preparation Zones

0 1 2.38

SystemElement 7Ingredients Prep

1 999 2.38

Submode 301Food and DrinkPreparation

0 1 2.13

Category 105Specialty Equipment

0 1 0.31

SystemElement 8Blast Chiller

1 700 0.02

SystemElement 9Wood Burning Oven

1 300 1.00

Category 106Preparation Protocol

0 1 2.29

SystemElement 10Food RecipeClipboards

1 500 2.17

SystemElement 11Kitchen ProcessSys-Clipboards

1 500 2.41

Category 107Plating/FinishingArea

0 1 1.08

SystemElement 12Plating Zone

1 999 1.08

Category 203Staging Meals

0 1 2.03

Category 108Service Circulation

0 1 3.04

SystemElement 13Expeditors

1 600 3.48

SystemElement 14POS Order Printers

1 400 2.38

Category 109Kitchen Communi-cation Protocol

0 1 1.96

SystemElement 15Employee Hierarchy

1 550 3.49

SystemElement 16BilingualCommunications

1 450 0.08

Category 204Sanitizing

0 1 1.35

Category 110Cleaning Equipment

0 1 1.63

SystemElement 17Heated Cleaning forGlassware

0 550 2.08

SystemElement 18Designated PotCleaning Area

0 450 1.07

Category 111Sanitation Procedure

0 1 1.08

SystemElement 19Mandatory CleaningProcess

0 999 1.08

Category 208Creating Ambiance

0 1 2.60

Category 122Sound Control

0 1 2.06

SystemElement 32Sheer Walls

0 500 2.05

SystemElement 33Ceiling Treatments

0 500 2.08

Submode 303Space Optimization

0 1 2.19

Category 123Impression-creatingDecor

0 1 2.70

SystemElement 34Rustic Material andFinishes

1 450 3.08

SystemElement 35Wine SupportiveTheme

2 550 2.39

System Merit 501Osteria Via Stato

2.71

Category 124Air Quality Control

0 1 3.11

SystemElement 36Temperature ControlSystem

1 650 3.11

SystemElement 37All Non-SmokingAreas

0 350 3.11

Category 209Controlling SpaceArrangement

0 1 1.79

Category 125Floor Plan

0 1 0.92

SystemElement 38Maximize SeatingArrangements

0 999 0.92

Category 126Adaptive Furnishings

0 1 2.65

SystemElement 39Communal Tables

0 500 2.08

SystemElement 40Dual FunctionEquipment

1 600 3.13

Category 127Vendor Relationships

0 1 2.16

SystemElement 41Speciality FoodSuppliers

2 500 2.15

SystemElement 42Alcoholic beveragesup

2 500 2.16

Category 210Acquiring Materials

0 1 3.02

Category 128Storage

0 1 3.44

SystemElement 43Basement DryStorage

1 999 3.44

Category 129Inventory Protocol

0 1 3.47

SystemElement 44Clipboard SystemInventory

1 999 3.47

Category 130Training Program

0 1 3.38

SystemElement 45Extended OsteriaTraining

2 600 3.38

SystemElement 46Cross Training Tests

1 400 3.38

Category 211Managing Staff

0 1 3.41

Category 131Assess/ DevelopProgram

0 1 3.38

SystemElement 47Twice-a-Year Reviews

0 500 3.27

SystemElement 48Goal OrientedImprovement

1 500 3.48

Submode 304Routine ConditionsManagement

0 1 3.28

Category 132Management Policy

0 1 3.47

SystemElement 49Leye ManagemntPolicies

1 999 3.47

Experimental System Evaluation Project

Osteria Via StatoSystem Assessment

Figure 16B Sample System Assessment in Horizontal Format (Segment 3)

Figure 16 Sample System Assessment in Horizontal Format (Segment 1)

lavender ring is now superimposed over any crit-icality factor switched on.

3. The value 0.00 was sometimes ambiguouswhen given for Functions and System Elementsbecause it can be attained in more than one way.For 0.00’s attained by cancellation of equal posi-tive and negative values, the yellow 0.00 blocknow remains as the correct assessment. For0.00’s attained because no positive or negativescores were given, a red-flagged yellow blockwas introduced (see Figure 14) signaling a Func-tion or System Element either not served or notused. For 0.00’s attained by offsetting positive

and negative scores whose final value is so smallthat it rounds off to 0.00, the value is artificiallyadjusted to just show within the 3-place valuewindow. This allows it to be seen to be numeri-cally within its color region, in agreement withthe block color triggered by the actual value.

4. Using the same System Structure archetypefor multiple systems sometimes led to situationsin which there were no System Elements to fill acategory (usually because of different emphasesin the system’s context of use). A provision for aweight of 0 was added to allow System categor-ies to be marked as knowingly unfilled and notincluded in calculations for higher level categoryIn

stitu

te o

f Des

ign,

Illin

ois

Inst

itute

of T

echn

olog

y htt

p://w

ww.id

.iit.ed

u


Category 205Preparing forService

0 1 1.82

Category 112Service Supply

0 1 3.49

SystemElement 20Pre-ServiceStaff-Meet & Taste

0 999 3.49

Category 113Set/Clear Procdure

0 1 0.16

SystemElement 21Pre-Srvce TableSetting

0 999 0.16

Mode 401Operation

0 1 2.07

Category 114Inquiry ResponseSystem

0 1 3.16

SystemElement 22Opentable

0 999 3.16

Submode 302Service

0 1 1.90

Category 206AccommodatingCustomers

0 1 0.96

Category 115Parking Provision

0 1 2.06

SystemElement 23Valet Parking

0 999 2.06

Category 116Hosting/Waiting Area

0 1 -3.50

SystemElement 24Designated WaitingArea

1 999 -2.08

Category 117Seating Protocol

0 1 2.12

SystemElement 25Osteria SeatingProtocol

1 500 2.15

SystemElement 26Customer Information

0 500 2.08

Category 118Service Support

0 1 3.29

SystemElement 27Multiful SpecService Stat

0 999 3.29

Category 207Providing Service

0 1 2.92

Category 119Staff CommicationProcedure

0 1 3.30

SystemElement 28Detailed DailyDocumentation

1 999 3.30

Category 120CustomerCommunication Policy

0 1 2.17

SystemElement 29Customer RelationsProtocol

1 999 2.17

Category 121Mood Lighting

0 1 2.52

SystemElement 30Light Sys Controland Dimmer

1 750 3.00

SystemElement 31Singl Ligt BulbPendnt Lamps

0 250 1.06

Category 212Complying with Lawsand Codes

0 1 3.48

Category 133Safety Provisions

0 1 3.50

SystemElement 50Designated SafetyStaff

0 999 3.50

Category 134ComplianceProcedures

0 1 3.47

SystemElement 51DesignatedComplaint Staff

0 999 3.47

Mode 402System Support

0 1 3.34

Category 135Waste DisposalSystem

0 1 3.08

SystemElement 52Recycling Service

0 999 3.08

Category 213MaintainingEnvironment

0 1 3.21

Category 136Cleaning Protocol

0 1 3.47

SystemElement 53Cleaning Checklist

1 999 3.47

Category 137Maintenance Program

0 1 3.08

SystemElement 54InternalMaintenance System

0 400 3.08

SystemElement 55Specialty EquipmentMaintenance

0 600 3.08

Category 214Marketing

0 1 3.44

Category 138Marketing/Communications

0 1 3.38

SystemElement 56Public RelationsExpert

0 300 3.38

SystemElement 57Word of MouthMarketing

0 700 3.38

Submode 305Change Management

0 1 3.40

Category 139IdentityReinforcement

0 1 3.49

SystemElement 58Enoteca

2 600 3.48

SystemElement 59Late Night WineTastng

1 400 3.50

Category 215Adapting

0 1 3.37

Category 140Analysis Tools

0 1 3.24

SystemElement 60Dish Prototyping

1 600 3.08

SystemElement 61Customer FeedbackLeye

1 400 3.47

Category 141Response Mechanism

0 1 3.50

SystemElement 62Food QualityControl System

1 999 3.50

Figure 16A Sample System Assessment in Horizontal Format (Segment 2)

Figure 16C Sample System Assessment in Horizontal Format (Segment 4)

values. A green-flagged yellow block appears inthe System Assessment structure (see Figure 14).

5. The effects of negative scores were direct-ly observable in the assessment hierarchies whenthe color of blocks indicated a negative blockvalue. Where negative scores simply lowered theaggregate positive value, however, it was notclear that they were responsible. To correct this,colored "value discs" were introduced on posi-tively colored blocks where any negative scorehad been registered (see Figures 13 and 14).

Summary and Conclusions

Hierarchical system evaluation enables new le-vels of access to system performance. From theexperiments reported in this paper, several in-sights can be drawn.

On the Good Side:1. Hierarchical description of criteria permits

analysis to be performed at levels of detail con-venient to purpose. Function Structure methodol-ogy from Structured Planning provides a soundbasis for both developing a structure and cover-ing system functionality thoroughly.In

stitu

te o

f Des

ign,

Illin

ois

Inst

itute

of T

echn

olog

y htt

p://w

ww.id

.iit.ed

u

Janu

ary,

2007


2. Hierarchical description of the system asan archetypal framework similarly supports fullcoverage of the system as the object of evalua-tion. Subject to purpose and available human re-sources, coverage can be extended to as fine alevel of detail as desired.

3. Measuring Functions or Defining State-ments against all System Elements, and measur-ing System Elements against all Functions orDefining Statements provides a degree of com-prehensiveness difficult to attain with other eval-uation methods.

4. The series calculation method for estab-lishing multivariable values for Functions, Defin-ing Statements and System Elements returnsvalues matched both to the measurement scaleand an evaluation policy that regards componentquality to be best represented by single strongestexpressions of support or obstruction.

5. An evaluation model combining a stan-dardized Function Structure with a System Struc-ture archetype designed for specific instantiationis a powerful system comparison tool. Thestrengths and weaknesses of systems can be ob-served and compared at equivalent positions andat parallel levels of abstraction.

6. Weighting of nodes within hierarchical ca-tegories provides a mechanism for matching cri-teria and system descriptions to context, givingan analyst the flexibility to fit a standardizedmodel to the subtleties of systems designed forniche environments.

On the Cautionary Side:1. While it is true for any evaluation tech-

nique, for this process it is more important thanever that evaluators have thorough knowledge oftheir field. The breadth of coverage required andlevel of detail possible demand in-depth domainknowledge and experience for success. A teamapproach offers a good solution; a panel or teamof specialists not only would increase the accura-cy of judgment, but would reduce the individualscoring load (see below).

2. Many scoring decisions must be made.The usefulness of an evaluation is strongly relat-ed to the level of detail covered; to a certain ex-tent, the more detail, the more thorough andbetter will be the results. For the three studiesreported in this paper, the number of scoring de-cisions for each restaurant was approximately6,000. For an evaluation with 150 criteria and

100 System Elements, the number would be15,000. Surprisingly, the process is not as timeconsuming as might be expected. The scores, byfar, are neutral "no effect"—as should beexpected—and are readily recognized as such.But scoring does take some hours. All computa-tion and graphic production processes (on thegood side) are done by the SYSEVAL computerprogram.

Acknowledgments

The author thanks the Lettuce Entertain You En-terprises organization and the staffs of its restau-rants for their support. In particular, thanks go toJohn Buchanan, President of the Lettuce Consult-ing Group, Mark Palicki, General Manager ofShaw’s Crabhouse; Sarah Johnson, now GeneralManager of VTK; Ed Culleeney, General Man-ager of Ben Pao; and Randy Brand, GeneralManager of Osteria Via Stato.

Nine graduate students conducted the analy-ses in the test trials, developed the test data, andcontributed recommendations for process im-provements: Clinton Barth, Michael Beebe,Christian Bernard, Rebecca Frisch, Chunlun Lee,Taylor Lies, Rosa Manfredi, Kristina Marich andJason Ring.

Special thanks are owed to Amgraf, Inc.maker of the MECCA2000 publishing softwareused by the visualization component of the SY-SEVAL computer program. Its accuracy andflexibility made possible the precise graphics re-quired in building the assessment structures. Fi-nally, the management of Steelcase, Inc. is oweda debt of gratitude for asking the questions thatinitiated this research in the first place.

References

Bruno, P (Sep 3, 2004) Chicago Sun-Times,Weekend Plus p 22

Cross, N (2000) Engineering design meth-ods: strategies for product design (3d) Wiley,Chichester

Ostrofsky, B (1977) Design, planning, anddevelopment methodology Prentice-Hall, Engle-wood Cliffs

Owen, C L (1992) Context for creativity De-sign Studies Vol 13 No 3 pp 216-228

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu


Owen, C L (1993) A critical role for designtechnology Design Management Journal Vol 3No 2 pp 10-18

Owen, C L (2001a) Structured planning indesign: information-age tools for product devel-opment Design Issues Vol 17 No 1 pp 27-43

Owen, C L (2001b) Structured planning: ad-vanced planning for business, institutions andgovernment, Institute of Design, Illinois Instituteof Technology, Chicago

Owen, C L (2005) Structured planning: ad-vanced planning for business, institutions andgovernment: notes on the process with summary

pages and examples Institute of Design, IllinoisInstitute of Technology, Chicago

Pahl, G and Beitz, W (1996) Engineeringdesign: a systematic approach (2d) Springer-Verlag, London

Roozenburg, N F M and Eckels, J (1995)Product design. fundamentals and methodsWiley, Chichester

Saaty, T L and Vargas, L G (2000) Models,methods, concepts & applications of the analytichierarchy process Springer-Verlag, London

Woodson, T T (1966) Introduction to engi-neering design McGraw-Hill, New York

Inst

itute

of D

esig

n, Il

linoi

s In

stitu

te o

f Tec

hnol

ogy

http:/

/www

.id.iit

.edu

Janu

ary,

2007

evaluation of complex systems - iit institute of design · charles l. owen evaluation of complex...

Documents