honr why departmenaustin/austin.d/talk.d/failure.pdf · br e ak?, f ebruary 1999 1 mark a. austin...
TRANSCRIPT
INSTITUTE FOR SYSTEMS RESEARCH
Introduction to Systems Engineering
Principles : Why things break?
HONR 289K : Why things break?, February 1999 1
Mark A. Austin
Department of Civil Engineering and Institute for Systems Research,
University of Maryland, College Park
E-mail : [email protected]
WWW : http://www.isr.umd.edu/�austin
INSTITUTE FOR SYSTEMS RESEARCH
Introduction to Systems Engineeering
Principles
HONR 289K : Why things break?, February 1999 2
Table of Contents
� Our De�nition of Systems Engineering
{ The Systems Engineering Lifecycle; Economic Considerations
� Features of Modern-day Engineering Systems
{ System Size and Complexity; Sources of Di�culty in Development; Lessons Learned
� Strategies for Learning about Complex Systems
{ Human decision making processes; Bene�ts of simulation/virtual worlds
� Models of Engineering Systems Development
� Quality Assurance and Control
� Contingency and Disaster Recovery Planning
INSTITUTE FOR SYSTEMS RESEARCH
Our De�nition of Systems Engineering
HONR 289K : Why things break?, February 1999 3
Characteritics of Engineering
� Goal oriented. Engineering starts only with a de�ned physical end result.
� Human centered. Engineering derives its goals from human wants and needs and this
has to consider the human conditions and environment.
� Systematic. The discipline of engineering builds on techniques and methods, and it uses
processes.
� Creative. Engineering changes the human environment by creating new systems.
Reference. Thom B., Systems Engineering : Principles and Practices of Computer-Based
Systems Engineering, John Wiley, 1993.
INSTITUTE FOR SYSTEMS RESEARCH
Our De�nition of Systems Engineering
HONR 289K : Why things break?, February 1999 4
HARDWARE ELEMENTS
SOFTWARE ELEMENTS
HUMAN ELEMENTSCONSTRAINTS.
SYSTEMS REQUIREMENTS ,
SPECIFICATIONS, AND
......................
.............
...............
ENVIRONMENT
OPERATIONAL
SYSTEMS
ENGINEERING
The goals of Systems Engineering are to provide:
� A balanced and disciplined approach to the total integration of the system building
blocks with the surrounding environment.
� A methodology for systems development that focussed on objectives, measurement,
and accomplishment.
� A systematic means to acquire information, and sort out and identify areas for trade-o�s
in cost, performance, reliability, quality ..etc.
INSTITUTE FOR SYSTEMS RESEARCH
Our De�nition of Systems Engineering
HONR 289K : Why things break?, February 1999 5
Traditional Engineering versus Systems Engineering
ENGINEERING COMPUTER SOFTWARE
AND HARDWARE.
BUSINESS
Breadth
LIASON AMONG DISCIPLINES
Dis
cipl
ine
Dep
th
DIS
CIP
LIN
ES
LIA
SO
N W
ITH
LIA
SO
N W
ITH
DIS
CIP
LIN
ES
LIA
SO
N W
ITH
DIS
CIP
LIN
ES
Modeling and
Simulation.... etc.....
Finance, Accounting....
Strategic Planning ....
Systems Tools .......
Networking .......
ANALYSIS AND TRADE - OFF STUDIES
SYSTEMS ENGINEERING
INSTITUTE FOR SYSTEMS RESEARCH
Our De�nition of Systems Engineering
HONR 289K : Why things break?, February 1999 6
CUSTOMER / USERS
ORGANIZATION REQUIREMENTS ENGINEERING
SYSTEM / PRODUCT-- Organization
Structure
-- Resources
-- Staff
-- Requirements
Capture
-- Req. Management
-- Traceability
-- Modeling /. Simulation
-- Maintenance
-- Design
ORGANIZATIONAL LEVEL PROJECT LEVEL
The key aspects of systems engineering are:
� Lifecycle Orientation. Design and development activities should consider all phases of
product creation, implementation, testing, maintenance and eventual retirement.
� Frontend Analysis of Requirements. Acquisition and (tradeo�) analysis of require-
ments, particularly in the early phases of the product lifecycle.
� Team Development. Coordination of activities across disciplines.
INSTITUTE FOR SYSTEMS RESEARCH
End-to-End Development of
Engineering Systems
HONR 289K : Why things break?, February 1999 7
Viewpoints of Design
� Design is a technical process to be accomplished. Technology is equipment and tools
that augment human capabilities and enrich the space of actions that can be taken. This
viewpoint focuses on individual designs.
� Design is an organizational process to be managed. The main job is coordination
(i.e., communication and bargaining) among separate enterprises (or groups within an
enterprise). Each enterprise/organization will have its own objectives, values, and polices.
This viewpoint focuses on the group.
Note. Organizations place more importance on the \process of creating the product" than
details of the product itself because organizational structures must support an ensemble of
design project developments, each with its own idiosyncrasies.
INSTITUTE FOR SYSTEMS RESEARCH
End-to-End Development of
Engineering Systems
HONR 289K : Why things break?, February 1999 8
SYSTEM DESIGN
MAINTENANCE
PLANNING IMPLEMENTATION
DESIGN AND
BUILD PHASE
Key steps in the lifecycle of an engineering system are:
� Translate a customer's business needs into system requirements.
� Systems design followed by detailed design.
� System Implementation. Integration and testing.
� Maintenance; planning for system upgrades.
� System disposal/retirement.
INSTITUTE FOR SYSTEMS RESEARCH
End-to-End Development of
Engineering Systems
HONR 289K : Why things break?, February 1999 9
CUSTOMER
CUSTOMER INTEFACE
CAPTURE
REQUIREMENTS ANALYSIS
AND DESIGN
IMPLEMENT TEST MAINTENANCE
DOCUMENTATION
PROJECT METRICS
PROJECT MANAGEMENT
CONFIGURATION MANAGEMENT / BASELINE CONTROL
INSTITUTE FOR SYSTEMS RESEARCH
End-to-End Development of
Engineering Systems
HONR 289K : Why things break?, February 1999 10
Approach to Problem Solving...
� Can a satisfactory functional design (what to do?) be realized by an available technical
solution (how to do it?) for an acceptable price (who to do it?).
Observation.
As we will soon see, too often system deliveries are:
� Behind schedule and over cost.
� Fail to meet the customers requirements,
� So unreliable that constant maintenance is needed.
� Poorly structured systems are di�cult to enhance and maintain.
INSTITUTE FOR SYSTEMS RESEARCH
End-to-End Development of
Engineering Systems
HONR 289K : Why things break?, February 1999 11
What versus How
SYSTEM DESCRIPTION
BEHAVIOR DESCRIPTION STRUCTURE DESCRIPTION
PARTS LISTCLASSIFICATION
DESCRIPTION
ASSEMBLY
IBUTES AND FUNC.
OBJECTS : ATTR-
DESCRIPTION
STATEFUNCTIONALDESCRIPTION
WHAT WILL THE SYSTEM DO ? HOW WILL THE SYSTEM DO IT ?
INSTITUTE FOR SYSTEMS RESEARCH
Routine versus Innovative
Development of Engineering Systems
HONR 289K : Why things break?, February 1999 12
INNOVATIVE
ROUTINE
TYPE OF DESIGN
COLLECT IDEAS
ORGANIZE INFORMATION
SHARE INFORMATION
DESIGN SYNTHESIS
DESIGNS
REUSE OF
IMPLEMENTATION
FULL - SCALEPROTOTYPE
Knowledge
Design
� Routine. Underlying procedures and technology are well tested and familiar. Few sur-
prises expected!
� Innovative. A more challenging problem because less information is known in the early
stages of development.
INSTITUTE FOR SYSTEMS RESEARCH
Post-delivery Failure Rates
HONR 289K : Why things break?, February 1999 13
Fai
lure
Rat
e
Lifetime
Phase 1
Infant Mortality Phase 2 - Normal Failure
Phase 3
Wornout Failure
Phases of Failure
� Infancy Operations. Defective components fail quickly, or system fails because of user
misuse.
� Normal Operations. Constant rate of failure.
� Aging Operations. Accelerated rate of failure due to aging of system/components.
INSTITUTE FOR SYSTEMS RESEARCH
Development of Engineering Systems :
Economic Considerations
HONR 289K : Why things break?, February 1999 14
75
50
25
100
Cum
ulat
ive
Per
cent
age Define Concept Commence Production
Funds Expended
Funds Committed
Product Lifecycle
Figure 1: Funding Commitments in Product Life-Cycle
INSTITUTE FOR SYSTEMS RESEARCH
Development of Engineering Systems :
Economic Considerations
HONR 289K : Why things break?, February 1999 15
Points to note are as follows:
� Decisions made early in the product life cycle require little expenditure, but commit the
largest proportion of project funds.
� Decisions made early in the life cycle have the largest opportunity for leading to signi�cant
cost savings. Conversely, decisions make in the latter stages of a product life cycle will
have little bearing on project cost (most of the money has already been spent!).
Economic constraints
Economic constraints limit the people, money, and tools resources which can be used during
the requirements engineering process.
INSTITUTE FOR SYSTEMS RESEARCH
Development of Engineering Systems :
Economic Considerations
HONR 289K : Why things break?, February 1999 16
Cost of Correcting Design Errors
Project Phase Bug Description Relative Cost
Design Design Team 1
Write and Test Designer 10-20
Quality Assurance QA Personnel 70-100
Shipment to Customer Customer Very-expensive
This table shows the results of a study by Hewlett-Packard to quantify the relative costs, in
terms of money and/or time, of correcting design errors. From an economic point-of-view, the
last thing a manufacturer wants is discovery of a fatal error in the engineering system by a
customer! Similar results are reported by IBM.
INSTITUTE FOR SYSTEMS RESEARCH
Development of Engineering Systems :
Economic Considerations
HONR 289K : Why things break?, February 1999 17
PRODUCT LIFE - CYCLE
CONCURRENT DESIGN
TRADITIONAL DESIGN
Commence Production
NU
MB
ER
OF
DE
SIG
N C
HA
NG
ES
Define Concept
Figure 2: Design Changes in Product Life-Cycle
INSTITUTE FOR SYSTEMS RESEARCH
Relative Lifecycle Costs for various
Engineering Systems
HONR 289K : Why things break?, February 1999 18
AEROSPACE AND DEFENSE
SOFTWARE
MARKET GOODS
PRODUCTION / OPERATIONS FACILITY[ CIVIL INFRASTRUCTURE ].
OPERATE DISPOSEDEVELOP BUILD
Lifecycle Duration. Software, 1 - 20yrs; Market Goods, 1 - 2 yrs; Civil Infrastructure 30 -
50 yrs. Projects with extended lifecycle duration soon become technically obsolete.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 19
Need for Careful Planning
The development of modern-day engineering systems needs to be carefully planned because:
� They are often developed by teams of engineers - team members must be able to under-
stand one-another's work.
� Well organized systems are easier to maintain. A little extra time spent on planning the
layout of a system may be saved many times over when it comes time for maintenance
and upgrades.
� Engineering systems evolve through a series of versions/updates. The project participants
making updates are unlikely to have been on the original development team.
How we work through these steps depends to a large extent on whether the development is
routine or not.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 20
Large Size. A system can be large in the number of requirements that must be met, the
number of constituent parts, components, and subsystems, and in the functions it performs.
� NASA's Space Station has in excess of 1.5 million project requirements.
� The Boeing 777 has 132,500 uniquely engineered parts, including 3,000,000 fasteners.
� The Sony Playstation is built from application speci�c integrated chips (ASIC) containing
1 million transistors. The next generation of ASIC chips will pack up to 49 million
transistors or switches onto a chip no larger than a postage stamp.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 21
Large Number of Functions (e.g., US Army Corps of Engineers)
� The US Army Corps of Engineers is one of the world's largest public engineering, design,
and construction management agencies. It supports Department of Defense programs
nationally (including other US government agencies, and eligible foreign governments),
and Europe, the Middle East, Africa, and parts of the Former Soviet Union.
� The US Army is currently working on the development of next-generation information and
systems services to support its design and construction activities. Construction activities
include: engineering services and contingency planning to support US military operations,
and peacetime planning, design, and construction. Such a computer-based system will in-
tegrate models, tasks and processes needed for management of army facilities, and provide
support for \facility life cycle decision making" at various levels of management:
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 22
Large Number of Functions
� Project Level. Fundamenal problem { how to select facilities for project functional
requirements { they must satisfy time and budgetary constraints throughout the life cycle.
A framework for project communications will include support for:
� Collaborative work among project partners (e.g., Army; American and Foreign Gov-
ernment; the customer; and contractors).
� Fiscal accounting and reporting to high-level organization.
� Guidance for decision making will be cast in terms of facility performance, quality, and
facility life cycle cost control.
� Training { interactive training for project management .
� National/State Level. At the National/State levels, issues are decision making assis-
tance for maintenance, refurbishment, monitoring of current and projected expenditures,
allocation of budgets at the base level.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 23
System/Problem Complexity. System complexity includes things like advanced nonlinear
mathematical models of systems, and system structures that have many interfaces carrying
high tra�c across them.
� Researchers in the Departments of Mechanical Engineering and Surgery at Stanford are
working on the development of an integrated set of tools to solve blood ow problems,
and to test hypotheses regarding vascular disease. The near-term objective is a web-based
environment for computer-aided surgical planning, and the solution of nonlinear problems
containing in excess of 2 million �nite elements.
� Utility functions describing (human) consumer preferences tend to be nonlinear.
� When it is fully operational, NASA's multi-satellite Earth Observing System (EOS) will
collect, process, and generate somewhere between 2-3 terabytes of information/data per
day for 15 to 20 years. The result will be a database containing 2-3 petabytes (1 petabyte
= 1015 bytes).
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 24
Data Analsysis Gap. A basic problem that exists today is that NASA is collecting and
storing more information than it can process by traditional means.
VOLUME
TIME1970 1980 1990
SCIENTISTS
DATA
TECNOLOGY
DATA ANALYSIS
NEED FOR AUTOMATED
Simply working harder is not the answer because there will never be enough scientists and
statiiticians to analyze the data.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 25
Distributed Development/Global Marketplace.
Many U.S. companies and organizations have reengineered (or are in the process of re-engineering)
their business operations to compete in a global marketplace.
� Example. In response to competition from Airbus, Boeing completely re-engineers the
way it designs and produces commercial aircraft.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 26
Cost Incurred by Poor Quality
20 %
15 %
10 %
5 %
0 %
25 %
Defects per Million66,807 308,5373.4 233 6210Cos
t of
Poo
r Q
ualit
y ;
e.g.
, In
spec
tion,
Average U.S. Company
World-Class Company
Rew
ork,
War
rant
y C
osts
, Lo
st S
ales
etc
...
On average, U.S. companies incur approximately 66,800 defects per million opportunities,
leading to a loss of 15% of sales due to poor quality. GE has 35,000 defects per million
opportunities. A few companies (e.g., Motorola, Honda, Texas Instruments) have reduced
defects in their design/production processes to a few hundred per million opportunities.
INSTITUTE FOR SYSTEMS RESEARCH
Modern-day Engineering Systems :
Sources of Di�culty in Development
HONR 289K : Why things break?, February 1999 27
Almost Flawless Production : Six Sigma Design
Mean value Mean value
One standard deviation
Specification limit Specification limit
4 Sigma Design2 1/2 Sigma Design
One standard deviation
Production Goal
Minimize variation in production quality to maximize number of standard deviations of pro-
duction variation inside limits of acceptable production speci�cation.
INSTITUTE FOR SYSTEMS RESEARCH
Modern-day Engineering Systems :
Sources of Di�culty in Development
HONR 289K : Why things break?, February 1999 28
Defects for various levels of sigma design
----------------------------------------------
Sigma level Defects per million opportunities
----------------------------------------------
2 308,537
3 66,807
4 6,210
5 233
6 3.4
----------------------------------------------
How to reach six-sigma status?
Experience indicates that in order for a company to reach six-sigma status it must:
� Explicitly account for 6� in the design of its production processes.
� Educate employees so that they understand quality and statistical process control.
� Reward employees for locating and recording sources of defects.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 29
Extended Project Duration
� Most U.S. companies operate on a product life-cycle of perhaps 1 to 2 years.
� For projects having a duration of more than a decade, long-term management of re-
quirements, adoption of new technology, and changes to personnel and budgets become a
signi�cant challenge.
Example.
� A pivotal component in the US Global Change Research Program is NASA's Mission to
Planet Earth Project (MTPE). MTPE will employ space and ground-based measurement
systems for the monitoring of planet Earth on a global scale over a 15 year lifetime.
INSTITUTE FOR SYSTEMS RESEARCH
Sources of Di�culty in Development
of Modern-day Engineering Systems
HONR 289K : Why things break?, February 1999 30
Vague Mission.
The title speaks for itself.
Example.
� The June 1996 issue of Scienti�c American features NASA's Space Station. Sta� writer
for Scienti�c American, Tim Beardsley, writes \the $27-billion international space station
will not do many of the jobs once conceived for it. Industrial interest in it has ebbed.
Uncertainties in Russia's commitment jeopardize its mission. Next year NASA will start
building it anyway."
INSTITUTE FOR SYSTEMS RESEARCH
Modern-day Engineering Systems :
Lessons Learned
HONR 289K : Why things break?, February 1999 31
The following �ndings are based on observations of 17 projects in the manufacturing, telecom-
munications, consumer electronics, and aerospace industries with budgets in the range $4.5 -
17.0 million. Di�culties in systems engineering projects are most likely to occur from:
� The thin spread of application domain knowledge.
� Fluctuating and con icting requirements.
� Communication and coordination breakdowns.
With respect to communication and coordination breakdowns:
� Education of people is often neglected.
� Often customers do not understand cost and tradeo�s.
� Don't rely on tools to overcome team and organizational problems.
INSTITUTE FOR SYSTEMS RESEARCH
Modern-day Engineering Systems :
Lessons Learned
HONR 289K : Why things break?, February 1999 32
Organizational Concerns
� Can a better design process be designed? Better usually means improved quality, reduced
lifecycle costs, and shorter development times.
� What information is needed and when so that concurrent design of products and processes
can proceed rapidly and with con�dence?
� What design tools are needed to support those parts of what designers do that is truly
di�cult? Do these tools even exist?
� What questions would the tools answer, and what information would they need?
� What cultural, human, and organizational changes are needed to make the \better design
process" work?
INSTITUTE FOR SYSTEMS RESEARCH
Learning about Complex Systems :
The Knowledge Gap
HONR 289K : Why things break?, February 1999 33
75
50
25
100
Cum
ulat
ive
Per
cent
age Define Concept Commence Production
Product Lifecycle
Funds Committed
Funds Expended
Knowledge
Kno
wle
dge
Gap
Ease of change
We would like our knowledge of a system to follow the contour of funds committed....so, how
to close the knowledge gap by improving the way we learn about complex systems ???
INSTITUTE FOR SYSTEMS RESEARCH
Learning about Complex Systems
HONR 289K : Why things break?, February 1999 34
Two-loop Feedback Structure for Decision Making
Information FeedbackMake Decisions
Employ Decision Rules
Understand Problem Structure
Formulate Strategy
Real World
Mental Model of the Real World
-- Belief of basic cause-and-effect
relationships affecting the
system operation.
Information feedback is the data, information, or behavior we sense about the real world.
A mental model is a belief of basic cause-and-e�ect relationships underpinning the system
response.
INSTITUTE FOR SYSTEMS RESEARCH
Learning about Complex Systems
HONR 289K : Why things break?, February 1999 35
Two Loops of Information Feedback
� In the outer loop, sensed information contributes to formation of a mental model of basic
cause-and-e�ect mechanisms.
� The inner loop represents situations where information is sensed, but for some reason,
perhaps oversight, is not accounted for in the mental model.
Impediments to clear cut Decision Making
� Unknown structure of the real world, dynamic complexity, time delays, and an inability to
infer dynamics from cognitive maps and/or conduct controled experiments on the system
itself.
� For most humans, an ability to reason in accordance with scienti�c explanation is not
a prerequisite for forming mental models of complex systems. They rely on intution,
experience, and reinforcement of ideas they believe to be true.
INSTITUTE FOR SYSTEMS RESEARCH
Learning about Complex Systems
HONR 289K : Why things break?, February 1999 36
Experimental Results from Cognitive Sciences
� Learning can occur even when the mental model is far from perfect.
� More important prerequisites are that each link in the two-feedback looping structure be
intact, and that we are able to cycle around the loops quickly relative to the rate at which
changes in the real world system are taking place.
� The most favorable conditions for learning exist when we have an ability to:
{ Generate new candidate decision rules su�ciently di�erent from current procedures,
and
{ Recognize and reward those decisions that improve performance.
Reference. Sterman J.D., Learning In and About Complex Systems, System Dynamics Re-
view, 1994.
INSTITUTE FOR SYSTEMS RESEARCH
Learning about Complex Systems
HONR 289K : Why things break?, February 1999 37
Limitations of Blind Processes
� So-called \blind processes" require a special set of circumstances to work well. And even
then, performance improvements are not guaranteed even after many iterates.
Bene�t of Scienti�c Principles
� The standard way of overcoming these limitations is to integrate scienti�c principles into
the decision making process.
� The use of scienti�c principles reduces the uncertainty associated with the problem solu-
tion, and enabling signi�cant improvements in performance in far fewer iterates than is
possible via evolutionary processes.
INSTITUTE FOR SYSTEMS RESEARCH
Learning about Complex Systems
HONR 289K : Why things break?, February 1999 38
E�ective Methods for Learning In and About Complex Systems
� Tools to elicit participant knowledge, articulate and reference perceptions, and create
maps of the feedback structure of a problem from the perceptions.
� Simulation tools to assess the dynamics of these maps, and to test and evaluate new
policies.
� Methods to improve scienti�c reasoning strategies, and strengthen group process routines.
Systems approaches that fail in any of these dimensions are unlikely to prove useful in enhancing
the capabilities of individuals and organizations.
Bene�t of Simulation and Virtual Worlds
An e�ective way of mitigating may of these learning impediments is to insert a virtual world
(or formal simulation model) into the learning feedback structure.
INS
TIT
UT
E F
OR
SY
ST
EM
S R
ES
EA
RC
H
LearningaboutComplexSystems
HONR289K:Whythingsbreak?,February1999
39
Real World
Information Feedback
Virtual World
Real World
-- Selective Perception-- Missing Feedback-- Delay.-- Ambiguity
Make Decisions
Real World
Virtual World
Virtual World
Mental Models
-- Assess group behavior/dynamics-- Application of scientific reasoning-- Mapping of feedback structure
Strategy, Structure, Decision Rules
-- Use simulation to infer dynamicsof cognitive maps....
-- Known Structure-- Variable Level of Complexity-- Controlled Experiments
-- Complete Information-- Complete Accuracy-- Immediate Feedback.
-- Performance is goal.-- Imperfect implementation.
-- Learning is goal.-- Perfect implementation.
Data Sol’ns
INSTITUTE FOR SYSTEMS RESEARCH
Models of Engineering Systems
Development
HONR 289K : Why things break?, February 1999 40
ORGANIZATION REQUIREMENTS
CUSTOMER / USERS
ENGINEERING
SYSTEM / PRODUCT
REAL WORLD SPACE
MODELING SPACE
Data Sol’ns Data Sol’ns
-- Strategy
-- Business Processes
-- How do you work ?
-- Resources / Staff
-- Technical Performance-- Requirements Capture
-- Requirements Management
-- Traceability
-- Design / Assembly
-- Maintenance
-- Returement
ORGANIZATIONAL
LEVEL
PROJECT LEVEL
Qualitative Content
Quantitative Content
MIX OF STRUCTURED / UNSTRUCTURED INFORMATION
INSTITUTE FOR SYSTEMS RESEARCH
Organizational Level : Waterfall
Model of System Development
HONR 289K : Why things break?, February 1999 41
Requirements
Definition
System Design
Detailed Design
Integration and
Test
Operations
and Maintenance
DESIGN [ HOW ] BUILD [ WHO ]ANALYSIS [ WHAT ]
INSTITUTE FOR SYSTEMS RESEARCH
Organizational Level : Spiral Model of
System Development
HONR 289K : Why things break?, February 1999 42
Service
Plan Next Phase.
Plan Next Phase.
Plan Next Phase.
SIMULATION AND MODELING.
Operational
Prototype.
Design.
Detailed
Design.
Preliminary
Integration and Test.
REVIEW
Requirements Plan.
Life Cycle Plan.
Determine Objectives and Alternatives.
Objectives and Alternatives.
Risk Analysis.
Risk Analysis.
Risk Analysis.
Requirements
Validation.
Testing of
Components.
Prototype 2.
Prototype 1.
INSTITUTE FOR SYSTEMS RESEARCH
Organizational Level : SEI Capability
Maturity Model
HONR 289K : Why things break?, February 1999 43
LEVEL RISK
Level 5
Optimized
Level 4
Managed
FOCUS
Continuous
Improvement
Product and
Process Quality
KEY PROCESS AREA
Defect Prevention
Process Change Management
Technology Change Management
Quantitaticve Process Management
Software Quality Management
Level 3
Managed
Engineering
Process
Organization Process Focus
Organization Process Definition
Peer Reviews
Training Program
Intergroup Coordination
Software Product Engineering
Integrated Software Management
Project
Management
Level 2
Repeatable
Software Project Planning
Software Project Tracking
Software Subcontract Management
Software Quality Assurance
Software Configuration Management
Requirements Management
Level 1
Ad - Hoc
Quality
Risk
INSTITUTE FOR SYSTEMS RESEARCH
Project Level : Stepwise Re�nement
of Engineering Development
HONR 289K : Why things break?, February 1999 44
TIME
AB
STR
AC
TIO
N
Map
Map
Map
Map
Sample
Optimization
Sample
Optimization
Requirements
Detailed
Design
Preliminary
Design
INSTITUTE FOR SYSTEMS RESEARCH
Project Level : Searching for Design
Alternatives/Solutions
HONR 289K : Why things break?, February 1999 45
More Abstract
Successful
Design
Failure
Requirements
Unexplored
Concepts
Less Concrete
More Concrete
Less Abstract
backtracking
INSTITUTE FOR SYSTEMS RESEARCH
Project Level : Requirements
Engineering and Engineering Design
HONR 289K : Why things break?, February 1999 46
SPECIFICATION-- Problems
Requirements
-- Informal
-- Domain
Knowledge
AcquisitionKnowledge
Validation Verification
Design-- Product
-- Information
System
REQUIREMENTS ENGINEERING
DESIGN ENGINEERING
INSTITUTE FOR SYSTEMS RESEARCH
Project Level : Requirements
Traceability
HONR 289K : Why things break?, February 1999 47
Sub-System Level
System Level
Component
Module
LAYERS OF REQUIREMENTS DESIGN
Design Versions
Trace
Trace
Trace
Two Basic Concerns
� Underdesign. Are all of the requirements accounted for in the design?
� Overdesign. Are all of design features really needed?
INSTITUTE FOR SYSTEMS RESEARCH
Project Level : Evaluation/Ranking of
Design Alternatives
HONR 289K : Why things break?, February 1999 48
Set
Priorities Analysis
Sensitivity
best alternatives....
Selection of
Average Value
Tolerance Levels
Alternatives Environment
Modeling
Effectiveness
Measures of
INSTITUTE FOR SYSTEMS RESEARCH
Project Level : Evaluation/Ranking of
Design Alternatives
HONR 289K : Why things break?, February 1999 49
SCHEDULEQUALITYCOST
DESIGN A
DESIGN B
DESIGN C
ALTERNATIVE
DESIGN
DESIGN OBJECTIVES
Each cell of the matrix should contain the best value possible, and estimates of design sensitivity
and con�dence. Sensitivity estimates allow us to see how robust the solution is to perturbations.
Con�dence test { \how con�dent are you that the numbers re ect reality?"
INSTITUTE FOR SYSTEMS RESEARCH
Quality Assessment of Engineering
Systems
HONR 289K : Why things break?, February 1999 50
Quality improvement is a major business strategy. This has happened for a number of reasons:
� Increasing consumer awareness of quality and a strong quality-performance orientation of
cosumers.
� Product liability, especially in the U.S.
� Increasing market competition leaves less room for unnecessary overheads in labor, energy
consumption, and use of raw materials.
� Dramatic improvements in productivity through e�ective programs of quality engineering
(more on this in a moment).
INSTITUTE FOR SYSTEMS RESEARCH
Role of Feedback in Quality
Assessment
HONR 289K : Why things break?, February 1999 51
Inputs
Raw materials
Components
Sub-assemblies
Manufactured Product
Services
Output
Uncontrollable
Input Parameters
Controllable Input ParametersCharacteristics
Measurement evaluation
ControlFeedback
Production Process
Quality
Synthesis of the model requires attention to three issues:
� Which inputs a�ect the quality characteristics?
� What is the relationship between the inputs and the quality characteristics?
� What is the best way of adjusting the controllable input parameters to improve quality?
INSTITUTE FOR SYSTEMS RESEARCH
Phase Diagram of Engineering Quality
Methods
HONR 289K : Why things break?, February 1999 52
Experiments
Design of
Acceptance
Sampling
Control
Process
0.0
1.0
Maturity of Organization
Rol
e o
f A
pplic
atio
n
� Acceptance Sampling. Identify and discard products that fall outside upper/lower
performance speci�cations.
� Statistical Process Control. Monitor process and detect when changes to input are
needed.
� Design of Experiments. Systematically vary the controllable input parameters and
observe e�ect on output product parameters.
INSTITUTE FOR SYSTEMS RESEARCH
Systematic Reduction of Process
Variability
HONR 289K : Why things break?, February 1999 53
Lower Specification
Limit
Limit
Upper Specification
Process mean
value
Acceptance
Sampling
Statistical Process
Control Experiments
Design of
� Design of Experiments is particularly e�ective in reducing product variability, one reason
being its application in the early phases of the product lifecycle.
INSTITUTE FOR SYSTEMS RESEARCH
Contingency and Disaster Recovery
Planning
HONR 289K : Why things break?, February 1999 54
Contingency Planning versus Disaster Planning
A sound contingency and disaster recovery plan anticipates the general consequences of likely
emergenciesk, and provides resources for recovery.
The occurrence of an adverse event triggers the need to decide whether to invoke the protections
they a�ord.
� Contingency Plans. Usually implemented as a hot site, with systems, data, communi-
cations, software, installed and operational. Contingency site need to be close (but not
too close) to the site of the primary event.
Contingency plans often ignore the physical extent of a potential disaster (e.g., loss of
access to a community after a severe earthquake).
� Disaster Plans. A disaster recovery site is usually implemented as a cold site. Process-
ing systems and infrastructure elements (e.g., communications, power) are installed and
activated promptly.
INSTITUTE FOR SYSTEMS RESEARCH
Contingency and Disaster Recovery
Planning
HONR 289K : Why things break?, February 1999 55
Economic Considerations
� For most applications, the degree of protection a�orded by contingency and disaster re-
covery planning is essentially and economic decision.
� Generally speaking, the cost of maintaining a cold site is much less than a hot site.
� The cost of contingency/disaster recovery should be treated as an insurance expense.
Reference. Assessing Emergencies and Making Disaster Recovery Decisions. Paper submitted
to INCOSE Conference, Brighton, England, June 1999.