designing ultra large scale systems list
DESCRIPTION
P.W.Anderson states in his classic paper titled "More is Different" - The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe. He further states - The constructionist hypothesis breaks down when confronted with the twin difficulties of scale and complexity. The behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of simple extrapolation of the properties of a few particles. Instead, at each level of complexity entirely new properties appear, and the understanding of the new behaviors requires research (fundamental).Complex systems are characterized by:Extraordinary decentralizationInherently conflicting, unknowable and diverse requirementsContinuous evolution and deploymentHeterogeneous, inconsistent, and changing elementsErosion of people/system boundaryNormal failuresNew paradigms for acquisition and policyThus far our methods to confront complexity have been based on reductionism or analysis, determinism, dualism, correspondence theory of knowledge and rationality – analytical and logical thinking as we know it. They have worked well for us in the past and continue to drive our approaches to problem solving, change creation and innovation.However, the new age of innovation warrants newer methods to deal with complexity. These new methods are likely to be based on a deeper understanding of indeterminacy, non-linearity, chaos, adaptation, self-organization and distributed intelligence.Crafitti provides an integrated approach to Ultra-large scale systems design using the Lean Inventive Systems Thinking Framework.This was presented at the DesignFirst Conference 2008 held at Bangalore.TRANSCRIPT
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Designing Ultra Large Scale Systems- The LIST* Approach
Navneet Bhushan & Karthikeyan Iyer
Crafitti Consulting
Oct 17, 2008
* Lean Inventive Systems Thinking
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
More is Different – Scale is the New Frontier - I
Google’s Custom Built Server FarmsCurrent estimates put Google's server farm at around 450,000 machines - and they're still custom built, commodity-class x86 PCs, just like they were in 1999
Oct 17, 2008
We Are Building Bigger and Bigger Systems
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Oct 17, 2008
The Internet Capillary networkInter-disciplinary collaborations
High School Friendships
Complex system design diagram
More is Different – Scale is the New Frontier - II
Systems are Evolving into Bigger Systems
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
More is Different – Scale is the New Frontier - III
“The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe” Anderson, P.W., More is Different,
Science, Vol. 177, No. 4047, Aug. 4, 1972, pp. 393-396.
Oct 17, 2008
Scale Changes Everything!
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
More is Different – Scale is the New Frontier - IV
Oct 17, 2008
“The older is not always a reliable model for the newer, the smaller for the larger, or the simpler for the more complex…Making something greater than any existing thing necessarily involves going beyond experience.”Henry Petroski, Pushing the Limits: New Adventures in Engineering
Scale is not a linear extrapolation!
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
SEI ULS Study – 2006!
Oct 17, 2008
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Why we don’t know how to Design ULSS?
Oct 17, 2008
System Design - Present ApproachesAll conflicts must be resolved centrally and uniformly
Requirements can be known in advance and change slowly.
Tradeoff decisions will be stable.
Discrete Time System improvements
Effect of a change can be predicted sufficiently well.
Configuration information
Components and users are fairly homogeneous.
People are just users of the system.
Collective behavior of people is not of interest.
Social interactions are not relevant.
Failures will occur infrequently. Defects can be removed.
A prime contractor is responsible for system development,
operation, and evolution.
ULS CharacteristicsDecentralized Control
Inherently conflicting, unknowable,
and diverse requirements
Continuous evolution and
deployment
Heterogeneous, inconsistent, and
changing elements
Erosion of the people/system
boundary
Normal Failures
New paradigms for acquisition and
policy
?
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
ULSS Research Areas
Oct 17, 2008
ULS Systems
Research Area
Specific Sub-Areas
Human Interaction
• Context-Aware Assistive computing
• Understanding Users and Their Contexts
• Modeling Users and User Communities
• Fostering Non-Competitive Social
Collaboration
• Longevity
Computational
Emergence
• Algorithmic Mechanism Design
• Metaheuristics in Software Engineering
• Digital Evolution
Design
• Design of All Levels
• Design Spaces and Design rules
• Harnessing Economics to Promote Good
Design
• Design Representation and Analysis
• Assimilation
• Determining and Managing
Requirements
Computational
Engineering
• Expressive Representation Languages
• Scaled-Up Specification, Verification, and
Certification
• Computational Engineering for Analysis
and Design
Adaptive
System
Infrastructure
• Decentralized Production Management
• View-Based Evolution
• Evolutionary Configuration and
Deployment
• In Situ Control and Adaptation
Adaptable and
Predictable
System Quality
• Robustness, Adaptation, and Quality
Attributes
• Scale and Composition of Quality
Attributes
• Understanding People-Centric Quality
Attributes
• Enforcing Quality Requirements
• Security, Trust, and Resiliency
• Engineering Management at Ultra-
Large Scales
Policy,
Acquisition,
and
Management
• Policy Definition for ULS Systems
• Fast Acquisition for ULS Systems
• Management of ULS Systems
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
LEAN INVENTIVE SYSTEMS THINKING (LIST)
Oct 17, 2008
CLASSICAL REDUCTIONISM
Analysis, Determinism,
Dualism, Correspondence
theory of knowledge,
Rationality, Artificial
THE LIST
Learning, Discovery, Design,
Evolutionary, Experimental,
Integrative, Holistic, Non-
linear, Natural
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Elements of System Design
Oct 17, 2008
Designing ULS Systems
Needs
Function
Structure
Behavior
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Designing for Complex Needs
Oct 17, 2008
01234Sight
Sound
Smell
Taste
Touch
Linguistic
Musical
Logical
Spatial
Kinesthetic
Intra-…
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Powerful Functional Design – TRIZ approach
• Eliminate System Contradictions
• Move to a higher level along proven lines of system evolution
• Ideal Final Result
Multiple design alternatives and
paths (Which one to choose?)
Oct 17, 2008
Improving parameter
Worsening parameter
Increasing dynamism (flexibility)
Transition to higher level systems
Transition to micro level systems
Completeness (reducing human involvement)
Shortening of energy flow path
Increasing controllability
Harmonization of rhythms
Non-uniform evolution of sub-systems
Ideality Quotient =Benefits
(Cost + Harmful Effects)
TRIZ: Theory of Inventive Problem Solving
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Structure – Complexity and Centrality
Oct 17, 2008
Flexibility, Adaptability, Ability to evolve, change
Stability, Rigidity, Strength, Productivity, Efficiency
Stability - They are elements where the most critical functions are performed and cannot afford to fail
Unpredictability – They are not very well understood and are unpredictable, hence they are deliberately operating within strict constraints
Dependency – Too many other system elements are dependent on this element, hence it cannot be changed very easily and without pain
Insulation – They are not very well connected and therefore do not have an incentive to change or adapt to changes
Efficiency – The elements are optimally structured to perform certain functions as efficiently as possible
Why are structures rigid?
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
System Behavior
• Behavior of the “whole” as opposed to the “parts”
• Need for observation from a different plane
• Complex non-linear systems display macro behavior invisible when seen from inside the system
– Synchronization
– Chaos
– Balance
Oct 17, 2008
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
System Behavior –The tendency to synchronize
• Synchronization aids stability –centralized control difficult in complex non-linear systems
• Synchronization is learnt over time; complex non-linear systems need to be designed as “learning systems”
• Systemic synchronization is a result of distributed intelligence
• Synchronization follows simple rules at the sub-system level
• Synchronization happens around system rhythms or clocks
• Synchronization happens for a reason (beneficial outcome)
Oct 17, 2008
Arrhythmia and cessation
of breath
Child rhythm
of breath
Toy rhythm
of breath
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
System Behavior - Designing for Chaos
• Chaotic systems hide certain patterns of behavior called attractors.
• Complex non-linear systems display chaotic behavior and gravitate towards system attractors
• Attractors manifest across scale
• Attractors act as strong central rhythms or clocks.
• Strange Attractors
Oct 17, 2008
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Design – A Macro“Balancing” act
Air
Distribution, change
Fire
Fuel, Energy to run the system
Water
Vitality, Life, Growth - Ideas
Earth
Structure and raw material
Complex ULS System
Oct 17, 2008
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
The “Typical”Design Process
Oct 17, 2008
Always Reaching Local Optima
Creates a false sense of simplification of complexity
Introduces artificial delay
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
The Lean Design Approach –Set-Based Concurrent Engineering
Search for global optima
Elimination of the weakest
Slow convergence
Emergent design
Scope to incorporate systems thinking (needs, functions, structure and behavior) and inventive thinking (TRIZ)
Oct 17, 2008
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
The Design Process and the LIST framework
Oct 17, 2008
Mapping the
Design Space
Describe user needs
In case of multiple needs carry out needs
interdependency analysis
Find out key functions to be performed
Understand structural complexities
Understand behavioral complexities
Function dependency analysis to find out
interdependencies
Can some high level functions specific to
strengths of different teams be identified
Let each team explore the specifications,
needs, functions independent of each
other
Each team explore design tradeoffs
through simulations and their past
observations
Each team should come up with their sets
of different solutions within the functional
and performance needs of the product
• Problem Formulation and Analysis• Value Stream• Ideal Final Result (IFR)• Why-what hierarchy• Nine windows• Dependency Structure Matrix (DSM) • Function/Attribute Analysis• System Complexity Estimator (SCE)• S curve analysis• Vedic Inventive Principles• Contradictions – Technical/Physical• Trends of evolution
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Design Process and LIST continued . . .
Oct 17, 2008
Striving for Conceptual
Robustness
(Functional Team level)
Design should remain functional
after variations in its
environment
Vulnerability of system to
changes in the environment
should be minimized
Modularized Design with
standard components
IFR
AFD/Subversion Analysis
Robust Inventive System Design (RISD)
DSM
Integration by
Intersection
(System level)
How are the parts integrated to
meet at the point that will be
regarded best solution
Find out overlap of feasible
design spaces for each sub
component
Decisions about eliminating the
weak designs
Decision Dependency Matrices (DDM)
Analytic Hierarchy Process (AHP)
Technical Contradictions / Inventive
Principles
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Design Process and LIST continued . . .
Oct 17, 2008
Establish Feasibility
before
Commitment
Multiple concepts developed using
prototyping simulation
The infeasible ones will be rejected rest all
will continue to be developed
Decision theoretic principles
AHP
Closer to IFR
Conflict Handling Cooperative Conflict handling Which solution is closer to IFR?
DDM
AHP
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
Final Points
• Scale is the New Frontier – it changes everything!
• Existing approaches found insufficient for designing Ultra Large Scale Systems
• Need for designing learning, discovery, human-machine cohesiveness and failure absorption inside the ULSS
• We propose a new framework combining elements of ancient wisdom, modern complexity science, empirical theory of invention, practical experiences of striving excellence and holistic design principles
• Lean Inventive Systems Thinking (LIST) may help us to bridge the gap between current approaches and needs of ULSS
Oct 17, 2008
cr
af
tin
g i
nn
ov
at
ion
to
ge
th
er
c r a f t i n g i n n o v a t i o n t o g e t h e r
THANK YOU!
Oct 17, 2008
Lean Inventive Systems Thinking
Crafitti ConsultingCrafting innovation together . . .
www.crafitti.com
Navneet Bhushan ([email protected])
Karthikeyan Iyer ([email protected])