honr why departmenaustin/austin.d/talk.d/failure.pdf · br e ak?, f ebruary 1999 1 mark a. austin...

55
INSTITUTE FOR SYSTEMS RESEARCH

Upload: phungkien

Post on 29-Jul-2018

215 views

Category:

Documents


0 download

TRANSCRIPT

INSTITUTE FOR SYSTEMS RESEARCH

Introduction to Systems Engineering

Principles : Why things break?

HONR 289K : Why things break?, February 1999 1

Mark A. Austin

Department of Civil Engineering and Institute for Systems Research,

University of Maryland, College Park

E-mail : [email protected]

WWW : http://www.isr.umd.edu/�austin

INSTITUTE FOR SYSTEMS RESEARCH

Introduction to Systems Engineeering

Principles

HONR 289K : Why things break?, February 1999 2

Table of Contents

� Our De�nition of Systems Engineering

{ The Systems Engineering Lifecycle; Economic Considerations

� Features of Modern-day Engineering Systems

{ System Size and Complexity; Sources of Di�culty in Development; Lessons Learned

� Strategies for Learning about Complex Systems

{ Human decision making processes; Bene�ts of simulation/virtual worlds

� Models of Engineering Systems Development

� Quality Assurance and Control

� Contingency and Disaster Recovery Planning

INSTITUTE FOR SYSTEMS RESEARCH

Our De�nition of Systems Engineering

HONR 289K : Why things break?, February 1999 3

Characteritics of Engineering

� Goal oriented. Engineering starts only with a de�ned physical end result.

� Human centered. Engineering derives its goals from human wants and needs and this

has to consider the human conditions and environment.

� Systematic. The discipline of engineering builds on techniques and methods, and it uses

processes.

� Creative. Engineering changes the human environment by creating new systems.

Reference. Thom B., Systems Engineering : Principles and Practices of Computer-Based

Systems Engineering, John Wiley, 1993.

INSTITUTE FOR SYSTEMS RESEARCH

Our De�nition of Systems Engineering

HONR 289K : Why things break?, February 1999 4

HARDWARE ELEMENTS

SOFTWARE ELEMENTS

HUMAN ELEMENTSCONSTRAINTS.

SYSTEMS REQUIREMENTS ,

SPECIFICATIONS, AND

......................

.............

...............

ENVIRONMENT

OPERATIONAL

SYSTEMS

ENGINEERING

The goals of Systems Engineering are to provide:

� A balanced and disciplined approach to the total integration of the system building

blocks with the surrounding environment.

� A methodology for systems development that focussed on objectives, measurement,

and accomplishment.

� A systematic means to acquire information, and sort out and identify areas for trade-o�s

in cost, performance, reliability, quality ..etc.

INSTITUTE FOR SYSTEMS RESEARCH

Our De�nition of Systems Engineering

HONR 289K : Why things break?, February 1999 5

Traditional Engineering versus Systems Engineering

ENGINEERING COMPUTER SOFTWARE

AND HARDWARE.

BUSINESS

Breadth

LIASON AMONG DISCIPLINES

Dis

cipl

ine

Dep

th

DIS

CIP

LIN

ES

LIA

SO

N W

ITH

LIA

SO

N W

ITH

DIS

CIP

LIN

ES

LIA

SO

N W

ITH

DIS

CIP

LIN

ES

Modeling and

Simulation.... etc.....

Finance, Accounting....

Strategic Planning ....

Systems Tools .......

Networking .......

ANALYSIS AND TRADE - OFF STUDIES

SYSTEMS ENGINEERING

INSTITUTE FOR SYSTEMS RESEARCH

Our De�nition of Systems Engineering

HONR 289K : Why things break?, February 1999 6

CUSTOMER / USERS

ORGANIZATION REQUIREMENTS ENGINEERING

SYSTEM / PRODUCT-- Organization

Structure

-- Resources

-- Staff

-- Requirements

Capture

-- Req. Management

-- Traceability

-- Modeling /. Simulation

-- Maintenance

-- Design

ORGANIZATIONAL LEVEL PROJECT LEVEL

The key aspects of systems engineering are:

� Lifecycle Orientation. Design and development activities should consider all phases of

product creation, implementation, testing, maintenance and eventual retirement.

� Frontend Analysis of Requirements. Acquisition and (tradeo�) analysis of require-

ments, particularly in the early phases of the product lifecycle.

� Team Development. Coordination of activities across disciplines.

INSTITUTE FOR SYSTEMS RESEARCH

End-to-End Development of

Engineering Systems

HONR 289K : Why things break?, February 1999 7

Viewpoints of Design

� Design is a technical process to be accomplished. Technology is equipment and tools

that augment human capabilities and enrich the space of actions that can be taken. This

viewpoint focuses on individual designs.

� Design is an organizational process to be managed. The main job is coordination

(i.e., communication and bargaining) among separate enterprises (or groups within an

enterprise). Each enterprise/organization will have its own objectives, values, and polices.

This viewpoint focuses on the group.

Note. Organizations place more importance on the \process of creating the product" than

details of the product itself because organizational structures must support an ensemble of

design project developments, each with its own idiosyncrasies.

INSTITUTE FOR SYSTEMS RESEARCH

End-to-End Development of

Engineering Systems

HONR 289K : Why things break?, February 1999 8

SYSTEM DESIGN

MAINTENANCE

PLANNING IMPLEMENTATION

DESIGN AND

BUILD PHASE

Key steps in the lifecycle of an engineering system are:

� Translate a customer's business needs into system requirements.

� Systems design followed by detailed design.

� System Implementation. Integration and testing.

� Maintenance; planning for system upgrades.

� System disposal/retirement.

INSTITUTE FOR SYSTEMS RESEARCH

End-to-End Development of

Engineering Systems

HONR 289K : Why things break?, February 1999 9

CUSTOMER

CUSTOMER INTEFACE

CAPTURE

REQUIREMENTS ANALYSIS

AND DESIGN

IMPLEMENT TEST MAINTENANCE

DOCUMENTATION

PROJECT METRICS

PROJECT MANAGEMENT

CONFIGURATION MANAGEMENT / BASELINE CONTROL

INSTITUTE FOR SYSTEMS RESEARCH

End-to-End Development of

Engineering Systems

HONR 289K : Why things break?, February 1999 10

Approach to Problem Solving...

� Can a satisfactory functional design (what to do?) be realized by an available technical

solution (how to do it?) for an acceptable price (who to do it?).

Observation.

As we will soon see, too often system deliveries are:

� Behind schedule and over cost.

� Fail to meet the customers requirements,

� So unreliable that constant maintenance is needed.

� Poorly structured systems are di�cult to enhance and maintain.

INSTITUTE FOR SYSTEMS RESEARCH

End-to-End Development of

Engineering Systems

HONR 289K : Why things break?, February 1999 11

What versus How

SYSTEM DESCRIPTION

BEHAVIOR DESCRIPTION STRUCTURE DESCRIPTION

PARTS LISTCLASSIFICATION

DESCRIPTION

ASSEMBLY

IBUTES AND FUNC.

OBJECTS : ATTR-

DESCRIPTION

STATEFUNCTIONALDESCRIPTION

WHAT WILL THE SYSTEM DO ? HOW WILL THE SYSTEM DO IT ?

INSTITUTE FOR SYSTEMS RESEARCH

Routine versus Innovative

Development of Engineering Systems

HONR 289K : Why things break?, February 1999 12

INNOVATIVE

ROUTINE

TYPE OF DESIGN

COLLECT IDEAS

ORGANIZE INFORMATION

SHARE INFORMATION

DESIGN SYNTHESIS

DESIGNS

REUSE OF

IMPLEMENTATION

FULL - SCALEPROTOTYPE

Knowledge

Design

� Routine. Underlying procedures and technology are well tested and familiar. Few sur-

prises expected!

� Innovative. A more challenging problem because less information is known in the early

stages of development.

INSTITUTE FOR SYSTEMS RESEARCH

Post-delivery Failure Rates

HONR 289K : Why things break?, February 1999 13

Fai

lure

Rat

e

Lifetime

Phase 1

Infant Mortality Phase 2 - Normal Failure

Phase 3

Wornout Failure

Phases of Failure

� Infancy Operations. Defective components fail quickly, or system fails because of user

misuse.

� Normal Operations. Constant rate of failure.

� Aging Operations. Accelerated rate of failure due to aging of system/components.

INSTITUTE FOR SYSTEMS RESEARCH

Development of Engineering Systems :

Economic Considerations

HONR 289K : Why things break?, February 1999 14

75

50

25

100

Cum

ulat

ive

Per

cent

age Define Concept Commence Production

Funds Expended

Funds Committed

Product Lifecycle

Figure 1: Funding Commitments in Product Life-Cycle

INSTITUTE FOR SYSTEMS RESEARCH

Development of Engineering Systems :

Economic Considerations

HONR 289K : Why things break?, February 1999 15

Points to note are as follows:

� Decisions made early in the product life cycle require little expenditure, but commit the

largest proportion of project funds.

� Decisions made early in the life cycle have the largest opportunity for leading to signi�cant

cost savings. Conversely, decisions make in the latter stages of a product life cycle will

have little bearing on project cost (most of the money has already been spent!).

Economic constraints

Economic constraints limit the people, money, and tools resources which can be used during

the requirements engineering process.

INSTITUTE FOR SYSTEMS RESEARCH

Development of Engineering Systems :

Economic Considerations

HONR 289K : Why things break?, February 1999 16

Cost of Correcting Design Errors

Project Phase Bug Description Relative Cost

Design Design Team 1

Write and Test Designer 10-20

Quality Assurance QA Personnel 70-100

Shipment to Customer Customer Very-expensive

This table shows the results of a study by Hewlett-Packard to quantify the relative costs, in

terms of money and/or time, of correcting design errors. From an economic point-of-view, the

last thing a manufacturer wants is discovery of a fatal error in the engineering system by a

customer! Similar results are reported by IBM.

INSTITUTE FOR SYSTEMS RESEARCH

Development of Engineering Systems :

Economic Considerations

HONR 289K : Why things break?, February 1999 17

PRODUCT LIFE - CYCLE

CONCURRENT DESIGN

TRADITIONAL DESIGN

Commence Production

NU

MB

ER

OF

DE

SIG

N C

HA

NG

ES

Define Concept

Figure 2: Design Changes in Product Life-Cycle

INSTITUTE FOR SYSTEMS RESEARCH

Relative Lifecycle Costs for various

Engineering Systems

HONR 289K : Why things break?, February 1999 18

AEROSPACE AND DEFENSE

SOFTWARE

MARKET GOODS

PRODUCTION / OPERATIONS FACILITY[ CIVIL INFRASTRUCTURE ].

OPERATE DISPOSEDEVELOP BUILD

Lifecycle Duration. Software, 1 - 20yrs; Market Goods, 1 - 2 yrs; Civil Infrastructure 30 -

50 yrs. Projects with extended lifecycle duration soon become technically obsolete.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 19

Need for Careful Planning

The development of modern-day engineering systems needs to be carefully planned because:

� They are often developed by teams of engineers - team members must be able to under-

stand one-another's work.

� Well organized systems are easier to maintain. A little extra time spent on planning the

layout of a system may be saved many times over when it comes time for maintenance

and upgrades.

� Engineering systems evolve through a series of versions/updates. The project participants

making updates are unlikely to have been on the original development team.

How we work through these steps depends to a large extent on whether the development is

routine or not.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 20

Large Size. A system can be large in the number of requirements that must be met, the

number of constituent parts, components, and subsystems, and in the functions it performs.

� NASA's Space Station has in excess of 1.5 million project requirements.

� The Boeing 777 has 132,500 uniquely engineered parts, including 3,000,000 fasteners.

� The Sony Playstation is built from application speci�c integrated chips (ASIC) containing

1 million transistors. The next generation of ASIC chips will pack up to 49 million

transistors or switches onto a chip no larger than a postage stamp.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 21

Large Number of Functions (e.g., US Army Corps of Engineers)

� The US Army Corps of Engineers is one of the world's largest public engineering, design,

and construction management agencies. It supports Department of Defense programs

nationally (including other US government agencies, and eligible foreign governments),

and Europe, the Middle East, Africa, and parts of the Former Soviet Union.

� The US Army is currently working on the development of next-generation information and

systems services to support its design and construction activities. Construction activities

include: engineering services and contingency planning to support US military operations,

and peacetime planning, design, and construction. Such a computer-based system will in-

tegrate models, tasks and processes needed for management of army facilities, and provide

support for \facility life cycle decision making" at various levels of management:

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 22

Large Number of Functions

� Project Level. Fundamenal problem { how to select facilities for project functional

requirements { they must satisfy time and budgetary constraints throughout the life cycle.

A framework for project communications will include support for:

� Collaborative work among project partners (e.g., Army; American and Foreign Gov-

ernment; the customer; and contractors).

� Fiscal accounting and reporting to high-level organization.

� Guidance for decision making will be cast in terms of facility performance, quality, and

facility life cycle cost control.

� Training { interactive training for project management .

� National/State Level. At the National/State levels, issues are decision making assis-

tance for maintenance, refurbishment, monitoring of current and projected expenditures,

allocation of budgets at the base level.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 23

System/Problem Complexity. System complexity includes things like advanced nonlinear

mathematical models of systems, and system structures that have many interfaces carrying

high tra�c across them.

� Researchers in the Departments of Mechanical Engineering and Surgery at Stanford are

working on the development of an integrated set of tools to solve blood ow problems,

and to test hypotheses regarding vascular disease. The near-term objective is a web-based

environment for computer-aided surgical planning, and the solution of nonlinear problems

containing in excess of 2 million �nite elements.

� Utility functions describing (human) consumer preferences tend to be nonlinear.

� When it is fully operational, NASA's multi-satellite Earth Observing System (EOS) will

collect, process, and generate somewhere between 2-3 terabytes of information/data per

day for 15 to 20 years. The result will be a database containing 2-3 petabytes (1 petabyte

= 1015 bytes).

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 24

Data Analsysis Gap. A basic problem that exists today is that NASA is collecting and

storing more information than it can process by traditional means.

VOLUME

TIME1970 1980 1990

SCIENTISTS

DATA

TECNOLOGY

DATA ANALYSIS

NEED FOR AUTOMATED

Simply working harder is not the answer because there will never be enough scientists and

statiiticians to analyze the data.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 25

Distributed Development/Global Marketplace.

Many U.S. companies and organizations have reengineered (or are in the process of re-engineering)

their business operations to compete in a global marketplace.

� Example. In response to competition from Airbus, Boeing completely re-engineers the

way it designs and produces commercial aircraft.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 26

Cost Incurred by Poor Quality

20 %

15 %

10 %

5 %

0 %

25 %

Defects per Million66,807 308,5373.4 233 6210Cos

t of

Poo

r Q

ualit

y ;

e.g.

, In

spec

tion,

Average U.S. Company

World-Class Company

Rew

ork,

War

rant

y C

osts

, Lo

st S

ales

etc

...

On average, U.S. companies incur approximately 66,800 defects per million opportunities,

leading to a loss of 15% of sales due to poor quality. GE has 35,000 defects per million

opportunities. A few companies (e.g., Motorola, Honda, Texas Instruments) have reduced

defects in their design/production processes to a few hundred per million opportunities.

INSTITUTE FOR SYSTEMS RESEARCH

Modern-day Engineering Systems :

Sources of Di�culty in Development

HONR 289K : Why things break?, February 1999 27

Almost Flawless Production : Six Sigma Design

Mean value Mean value

One standard deviation

Specification limit Specification limit

4 Sigma Design2 1/2 Sigma Design

One standard deviation

Production Goal

Minimize variation in production quality to maximize number of standard deviations of pro-

duction variation inside limits of acceptable production speci�cation.

INSTITUTE FOR SYSTEMS RESEARCH

Modern-day Engineering Systems :

Sources of Di�culty in Development

HONR 289K : Why things break?, February 1999 28

Defects for various levels of sigma design

----------------------------------------------

Sigma level Defects per million opportunities

----------------------------------------------

2 308,537

3 66,807

4 6,210

5 233

6 3.4

----------------------------------------------

How to reach six-sigma status?

Experience indicates that in order for a company to reach six-sigma status it must:

� Explicitly account for 6� in the design of its production processes.

� Educate employees so that they understand quality and statistical process control.

� Reward employees for locating and recording sources of defects.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 29

Extended Project Duration

� Most U.S. companies operate on a product life-cycle of perhaps 1 to 2 years.

� For projects having a duration of more than a decade, long-term management of re-

quirements, adoption of new technology, and changes to personnel and budgets become a

signi�cant challenge.

Example.

� A pivotal component in the US Global Change Research Program is NASA's Mission to

Planet Earth Project (MTPE). MTPE will employ space and ground-based measurement

systems for the monitoring of planet Earth on a global scale over a 15 year lifetime.

INSTITUTE FOR SYSTEMS RESEARCH

Sources of Di�culty in Development

of Modern-day Engineering Systems

HONR 289K : Why things break?, February 1999 30

Vague Mission.

The title speaks for itself.

Example.

� The June 1996 issue of Scienti�c American features NASA's Space Station. Sta� writer

for Scienti�c American, Tim Beardsley, writes \the $27-billion international space station

will not do many of the jobs once conceived for it. Industrial interest in it has ebbed.

Uncertainties in Russia's commitment jeopardize its mission. Next year NASA will start

building it anyway."

INSTITUTE FOR SYSTEMS RESEARCH

Modern-day Engineering Systems :

Lessons Learned

HONR 289K : Why things break?, February 1999 31

The following �ndings are based on observations of 17 projects in the manufacturing, telecom-

munications, consumer electronics, and aerospace industries with budgets in the range $4.5 -

17.0 million. Di�culties in systems engineering projects are most likely to occur from:

� The thin spread of application domain knowledge.

� Fluctuating and con icting requirements.

� Communication and coordination breakdowns.

With respect to communication and coordination breakdowns:

� Education of people is often neglected.

� Often customers do not understand cost and tradeo�s.

� Don't rely on tools to overcome team and organizational problems.

INSTITUTE FOR SYSTEMS RESEARCH

Modern-day Engineering Systems :

Lessons Learned

HONR 289K : Why things break?, February 1999 32

Organizational Concerns

� Can a better design process be designed? Better usually means improved quality, reduced

lifecycle costs, and shorter development times.

� What information is needed and when so that concurrent design of products and processes

can proceed rapidly and with con�dence?

� What design tools are needed to support those parts of what designers do that is truly

di�cult? Do these tools even exist?

� What questions would the tools answer, and what information would they need?

� What cultural, human, and organizational changes are needed to make the \better design

process" work?

INSTITUTE FOR SYSTEMS RESEARCH

Learning about Complex Systems :

The Knowledge Gap

HONR 289K : Why things break?, February 1999 33

75

50

25

100

Cum

ulat

ive

Per

cent

age Define Concept Commence Production

Product Lifecycle

Funds Committed

Funds Expended

Knowledge

Kno

wle

dge

Gap

Ease of change

We would like our knowledge of a system to follow the contour of funds committed....so, how

to close the knowledge gap by improving the way we learn about complex systems ???

INSTITUTE FOR SYSTEMS RESEARCH

Learning about Complex Systems

HONR 289K : Why things break?, February 1999 34

Two-loop Feedback Structure for Decision Making

Information FeedbackMake Decisions

Employ Decision Rules

Understand Problem Structure

Formulate Strategy

Real World

Mental Model of the Real World

-- Belief of basic cause-and-effect

relationships affecting the

system operation.

Information feedback is the data, information, or behavior we sense about the real world.

A mental model is a belief of basic cause-and-e�ect relationships underpinning the system

response.

INSTITUTE FOR SYSTEMS RESEARCH

Learning about Complex Systems

HONR 289K : Why things break?, February 1999 35

Two Loops of Information Feedback

� In the outer loop, sensed information contributes to formation of a mental model of basic

cause-and-e�ect mechanisms.

� The inner loop represents situations where information is sensed, but for some reason,

perhaps oversight, is not accounted for in the mental model.

Impediments to clear cut Decision Making

� Unknown structure of the real world, dynamic complexity, time delays, and an inability to

infer dynamics from cognitive maps and/or conduct controled experiments on the system

itself.

� For most humans, an ability to reason in accordance with scienti�c explanation is not

a prerequisite for forming mental models of complex systems. They rely on intution,

experience, and reinforcement of ideas they believe to be true.

INSTITUTE FOR SYSTEMS RESEARCH

Learning about Complex Systems

HONR 289K : Why things break?, February 1999 36

Experimental Results from Cognitive Sciences

� Learning can occur even when the mental model is far from perfect.

� More important prerequisites are that each link in the two-feedback looping structure be

intact, and that we are able to cycle around the loops quickly relative to the rate at which

changes in the real world system are taking place.

� The most favorable conditions for learning exist when we have an ability to:

{ Generate new candidate decision rules su�ciently di�erent from current procedures,

and

{ Recognize and reward those decisions that improve performance.

Reference. Sterman J.D., Learning In and About Complex Systems, System Dynamics Re-

view, 1994.

INSTITUTE FOR SYSTEMS RESEARCH

Learning about Complex Systems

HONR 289K : Why things break?, February 1999 37

Limitations of Blind Processes

� So-called \blind processes" require a special set of circumstances to work well. And even

then, performance improvements are not guaranteed even after many iterates.

Bene�t of Scienti�c Principles

� The standard way of overcoming these limitations is to integrate scienti�c principles into

the decision making process.

� The use of scienti�c principles reduces the uncertainty associated with the problem solu-

tion, and enabling signi�cant improvements in performance in far fewer iterates than is

possible via evolutionary processes.

INSTITUTE FOR SYSTEMS RESEARCH

Learning about Complex Systems

HONR 289K : Why things break?, February 1999 38

E�ective Methods for Learning In and About Complex Systems

� Tools to elicit participant knowledge, articulate and reference perceptions, and create

maps of the feedback structure of a problem from the perceptions.

� Simulation tools to assess the dynamics of these maps, and to test and evaluate new

policies.

� Methods to improve scienti�c reasoning strategies, and strengthen group process routines.

Systems approaches that fail in any of these dimensions are unlikely to prove useful in enhancing

the capabilities of individuals and organizations.

Bene�t of Simulation and Virtual Worlds

An e�ective way of mitigating may of these learning impediments is to insert a virtual world

(or formal simulation model) into the learning feedback structure.

INS

TIT

UT

E F

OR

SY

ST

EM

S R

ES

EA

RC

H

LearningaboutComplexSystems

HONR289K:Whythingsbreak?,February1999

39

Real World

Information Feedback

Virtual World

Real World

-- Selective Perception-- Missing Feedback-- Delay.-- Ambiguity

Make Decisions

Real World

Virtual World

Virtual World

Mental Models

-- Assess group behavior/dynamics-- Application of scientific reasoning-- Mapping of feedback structure

Strategy, Structure, Decision Rules

-- Use simulation to infer dynamicsof cognitive maps....

-- Known Structure-- Variable Level of Complexity-- Controlled Experiments

-- Complete Information-- Complete Accuracy-- Immediate Feedback.

-- Performance is goal.-- Imperfect implementation.

-- Learning is goal.-- Perfect implementation.

Data Sol’ns

INSTITUTE FOR SYSTEMS RESEARCH

Models of Engineering Systems

Development

HONR 289K : Why things break?, February 1999 40

ORGANIZATION REQUIREMENTS

CUSTOMER / USERS

ENGINEERING

SYSTEM / PRODUCT

REAL WORLD SPACE

MODELING SPACE

Data Sol’ns Data Sol’ns

-- Strategy

-- Business Processes

-- How do you work ?

-- Resources / Staff

-- Technical Performance-- Requirements Capture

-- Requirements Management

-- Traceability

-- Design / Assembly

-- Maintenance

-- Returement

ORGANIZATIONAL

LEVEL

PROJECT LEVEL

Qualitative Content

Quantitative Content

MIX OF STRUCTURED / UNSTRUCTURED INFORMATION

INSTITUTE FOR SYSTEMS RESEARCH

Organizational Level : Waterfall

Model of System Development

HONR 289K : Why things break?, February 1999 41

Requirements

Definition

System Design

Detailed Design

Integration and

Test

Operations

and Maintenance

DESIGN [ HOW ] BUILD [ WHO ]ANALYSIS [ WHAT ]

INSTITUTE FOR SYSTEMS RESEARCH

Organizational Level : Spiral Model of

System Development

HONR 289K : Why things break?, February 1999 42

Service

Plan Next Phase.

Plan Next Phase.

Plan Next Phase.

SIMULATION AND MODELING.

Operational

Prototype.

Design.

Detailed

Design.

Preliminary

Integration and Test.

REVIEW

Requirements Plan.

Life Cycle Plan.

Determine Objectives and Alternatives.

Objectives and Alternatives.

Risk Analysis.

Risk Analysis.

Risk Analysis.

Requirements

Validation.

Testing of

Components.

Prototype 2.

Prototype 1.

INSTITUTE FOR SYSTEMS RESEARCH

Organizational Level : SEI Capability

Maturity Model

HONR 289K : Why things break?, February 1999 43

LEVEL RISK

Level 5

Optimized

Level 4

Managed

FOCUS

Continuous

Improvement

Product and

Process Quality

KEY PROCESS AREA

Defect Prevention

Process Change Management

Technology Change Management

Quantitaticve Process Management

Software Quality Management

Level 3

Managed

Engineering

Process

Organization Process Focus

Organization Process Definition

Peer Reviews

Training Program

Intergroup Coordination

Software Product Engineering

Integrated Software Management

Project

Management

Level 2

Repeatable

Software Project Planning

Software Project Tracking

Software Subcontract Management

Software Quality Assurance

Software Configuration Management

Requirements Management

Level 1

Ad - Hoc

Quality

Risk

INSTITUTE FOR SYSTEMS RESEARCH

Project Level : Stepwise Re�nement

of Engineering Development

HONR 289K : Why things break?, February 1999 44

TIME

AB

STR

AC

TIO

N

Map

Map

Map

Map

Sample

Optimization

Sample

Optimization

Requirements

Detailed

Design

Preliminary

Design

INSTITUTE FOR SYSTEMS RESEARCH

Project Level : Searching for Design

Alternatives/Solutions

HONR 289K : Why things break?, February 1999 45

More Abstract

Successful

Design

Failure

Requirements

Unexplored

Concepts

Less Concrete

More Concrete

Less Abstract

backtracking

INSTITUTE FOR SYSTEMS RESEARCH

Project Level : Requirements

Engineering and Engineering Design

HONR 289K : Why things break?, February 1999 46

SPECIFICATION-- Problems

Requirements

-- Informal

-- Domain

Knowledge

AcquisitionKnowledge

Validation Verification

Design-- Product

-- Information

System

REQUIREMENTS ENGINEERING

DESIGN ENGINEERING

INSTITUTE FOR SYSTEMS RESEARCH

Project Level : Requirements

Traceability

HONR 289K : Why things break?, February 1999 47

Sub-System Level

System Level

Component

Module

LAYERS OF REQUIREMENTS DESIGN

Design Versions

Trace

Trace

Trace

Two Basic Concerns

� Underdesign. Are all of the requirements accounted for in the design?

� Overdesign. Are all of design features really needed?

INSTITUTE FOR SYSTEMS RESEARCH

Project Level : Evaluation/Ranking of

Design Alternatives

HONR 289K : Why things break?, February 1999 48

Set

Priorities Analysis

Sensitivity

best alternatives....

Selection of

Average Value

Tolerance Levels

Alternatives Environment

Modeling

Effectiveness

Measures of

INSTITUTE FOR SYSTEMS RESEARCH

Project Level : Evaluation/Ranking of

Design Alternatives

HONR 289K : Why things break?, February 1999 49

SCHEDULEQUALITYCOST

DESIGN A

DESIGN B

DESIGN C

ALTERNATIVE

DESIGN

DESIGN OBJECTIVES

Each cell of the matrix should contain the best value possible, and estimates of design sensitivity

and con�dence. Sensitivity estimates allow us to see how robust the solution is to perturbations.

Con�dence test { \how con�dent are you that the numbers re ect reality?"

INSTITUTE FOR SYSTEMS RESEARCH

Quality Assessment of Engineering

Systems

HONR 289K : Why things break?, February 1999 50

Quality improvement is a major business strategy. This has happened for a number of reasons:

� Increasing consumer awareness of quality and a strong quality-performance orientation of

cosumers.

� Product liability, especially in the U.S.

� Increasing market competition leaves less room for unnecessary overheads in labor, energy

consumption, and use of raw materials.

� Dramatic improvements in productivity through e�ective programs of quality engineering

(more on this in a moment).

INSTITUTE FOR SYSTEMS RESEARCH

Role of Feedback in Quality

Assessment

HONR 289K : Why things break?, February 1999 51

Inputs

Raw materials

Components

Sub-assemblies

Manufactured Product

Services

Output

Uncontrollable

Input Parameters

Controllable Input ParametersCharacteristics

Measurement evaluation

ControlFeedback

Production Process

Quality

Synthesis of the model requires attention to three issues:

� Which inputs a�ect the quality characteristics?

� What is the relationship between the inputs and the quality characteristics?

� What is the best way of adjusting the controllable input parameters to improve quality?

INSTITUTE FOR SYSTEMS RESEARCH

Phase Diagram of Engineering Quality

Methods

HONR 289K : Why things break?, February 1999 52

Experiments

Design of

Acceptance

Sampling

Control

Process

0.0

1.0

Maturity of Organization

Rol

e o

f A

pplic

atio

n

� Acceptance Sampling. Identify and discard products that fall outside upper/lower

performance speci�cations.

� Statistical Process Control. Monitor process and detect when changes to input are

needed.

� Design of Experiments. Systematically vary the controllable input parameters and

observe e�ect on output product parameters.

INSTITUTE FOR SYSTEMS RESEARCH

Systematic Reduction of Process

Variability

HONR 289K : Why things break?, February 1999 53

Lower Specification

Limit

Limit

Upper Specification

Process mean

value

Acceptance

Sampling

Statistical Process

Control Experiments

Design of

� Design of Experiments is particularly e�ective in reducing product variability, one reason

being its application in the early phases of the product lifecycle.

INSTITUTE FOR SYSTEMS RESEARCH

Contingency and Disaster Recovery

Planning

HONR 289K : Why things break?, February 1999 54

Contingency Planning versus Disaster Planning

A sound contingency and disaster recovery plan anticipates the general consequences of likely

emergenciesk, and provides resources for recovery.

The occurrence of an adverse event triggers the need to decide whether to invoke the protections

they a�ord.

� Contingency Plans. Usually implemented as a hot site, with systems, data, communi-

cations, software, installed and operational. Contingency site need to be close (but not

too close) to the site of the primary event.

Contingency plans often ignore the physical extent of a potential disaster (e.g., loss of

access to a community after a severe earthquake).

� Disaster Plans. A disaster recovery site is usually implemented as a cold site. Process-

ing systems and infrastructure elements (e.g., communications, power) are installed and

activated promptly.

INSTITUTE FOR SYSTEMS RESEARCH

Contingency and Disaster Recovery

Planning

HONR 289K : Why things break?, February 1999 55

Economic Considerations

� For most applications, the degree of protection a�orded by contingency and disaster re-

covery planning is essentially and economic decision.

� Generally speaking, the cost of maintaining a cold site is much less than a hot site.

� The cost of contingency/disaster recovery should be treated as an insurance expense.

Reference. Assessing Emergencies and Making Disaster Recovery Decisions. Paper submitted

to INCOSE Conference, Brighton, England, June 1999.