yves caseau@md day2011
DESCRIPTION
TRANSCRIPT
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 1/31
Complexity, Modularity, Abstraction:
Designing Architecture-Oriented Services
Model-Driven Day
November 24th, 2011 (v0.0)
Yves Caseau
Bouygues Telecom – Académie des Technologies
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 2/31
Outline
1. Complexity
the 21st century challenge for enterprises and their information
system
2. Enterprise Architecture
Anticipation in a complex world is not forecasting,
but building a « situation potential »
3. SOA and sustainability
repeatable long-term performance requires discipline /
governance
4. Cloud-Ready Architecture
Distributed, scalable, massively parallel computing resource for
the whole information system
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 3/31
Companies are Facing a Complex World
A Complex World:
Hyper-competition, globalization, time is shrinking
The power has shifted to the consumers (F. Dupuy)
T. Friedman : « All that is easy has been done, what’s left is the hard
stuff »
Complicated problems require specialists,
Complex problems require everyone
Diversity of skills and viewpoints …
… organized into teams
Complex problems are solved “on the gemba”,
where they occur, one at a time
Abstractions hide too much, decomposition does not work !
“Reproducible conditions” … do not always exist (isolation is impossible)
Communication is hard (cf. IT when specifying is harder than coding)
Part
I:
Com
ple
xty
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 4/31
An Enterprise is a « Complex System »
Numerical complexity (number of things)
Multi-scale
Temporal complexity
Richness of interactions with environment
Symptom examples:
Costs (Information Systems evolution)
Example: non-linear evolution of project costs w.r.t. project size
Rate of errors & failures
Difficulty to « guarantee » robustness and fault-tolerance
Ross Ashby « regulation of a (complex) system requires a control system
that is as complex as the system itself »
Time-to-market
The first complexity-related issue
Time-to-integrate-a-new-component depends on the host size :
– Human complexity (organization)
– Modularity failure (unforseen impacts & interaction between components)
Law of unintended consequences – Feature Interaction Problem
Part
I:
Com
ple
xty
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 5/31
A systemic view of the enterprise & its IS
Decision System
Information System
IS
environnement
Operating
System
Precesses
Input flows
Structure
Finality
Evolution
Business Model (CEISAR)
Organisation (CEISAR)
Political Aspects
Objets
Changes
“Pragmatic” aspect
operation procedures
“pragma” aspect
Functional
Model Process
Model
Semantic aspect
Logic aspect
IS is a
complex
system itself
Software aspect
Hardware
Geographical
/ localization
Part
I:
Com
ple
xty
PRAXEM
E
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 6/31
Consequences of a « systemic vision »
Features / properties « emergence »
from « Performance by Design » …
.. to « Performance though simulation & prototypes »
Humility and Continuous improvement
Complexity Governance
Acknowledge the problem !
Attack it with method
Cf. CEISAR’s cube
(three dimensions)
Explicit policies are a key governance tools
SLA, service contacts, data ownership policies, …
“Enterprise Architecture” as a corporate practice
Stakeholders alignment
The “external environment” is critical to IS strategic planning
Complexité
Agilité
Synergie
Exécution
dans le
Monde Réel
Modèle
Eléments Spécifiques
Operations Transformations
Eléments Partageables
ou Réutilisables
Part
I:
Com
ple
xty
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 7/31
Measuring Complexity
Measuring the numerical complexity
Sizing the objects (TPMC, FP, Tb)
Complexity (« relationship intricacy ») Metrics
DSM (design structure matrix)
Cyclomatic complexity
Scalar Euclidian Complexity
The only scale-invariant metric that applies to an architecture map
Bottom line : metrics offer a small amount of help
Part
I:
Com
ple
xty
Gyx
yxG),(
)().(),(
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 8/31
Taming Information System Complexity (I)
Simple approach:
maps and architecture schemas (UML2 helps ).
Common sense approach:
Energetic standardization
Reducing heterogeneity reduces complexity (Printz).
Technological approach:
Infrastructure (middleware)
Introduction of physical de-coupling, sharing and intermediation
Part
I:
Com
ple
xty
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 9/31
Taming Information Systems Complexity (II)
Systematic approach (« Urbanisation »):
Enterprise Architecture, seen as a company-wide practice, since it helps to
align everyone towards a common target and a transformation path.
Architectural approach (hardest, at the structure level):
modularity (semantic de-coupling).
Strategic Approach (Service-Oriented Architecture):
SOA as a governance practice, reduces complexity:
share & reuse
Common understanding
Reduces the temporal complexity
Part
I:
Com
ple
xty
Cf. Parts
II & III
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 10/31
Part II
1. Complexity
the 21st century challenge for enterprises and their information
system
2. Enterprise Architecture
Anticipation in a complex world is not forecasting,
but building a « situation potential »
3. SOA and sustainability
repeatable long-term performance requires discipline /
governance
4. Cloud-ready architecture
Distributed, scalable, massively parallel computing resource for
the whole information system
Lessons from Bouygues Telecom
Back-office re-engineering 2001-2005
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 11/31
Enterprise Architecture
Architecture: why ? IS Re-engineering
Communicate a vision
Key transformation tool
Favor reuse
Reduce complexity and costs
Support standardization
Align stakeholders
Architects need to avoid
complex tools & formalisms
Sensitive to each
enterprise’s maturity level
Asynchronous / diachronous
Acts as a corporate memory
Visual change management
« Urbanisation »
Component-based
Process-oriented (business logic extraction)
Temporal de-coupling (asynchronous message)
Functional de-coupling (intermediation)
« Enterprise Architecture »
Coherence of three domains
Strategy : goals
operations: processes and data
Information Systems: applications and services
Similar topic / two views
EA is more concerned with the target
Re-engineering is a process
Part
II:
Arc
hit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 12/31
Data Architecture
Data Model
Semantic reference: what does a customer, a bill, … mean ?
Conceptual model: how is a customer defined ?
Ontology: class hierarchy (UML)
Major business asset & key alignment tool
Data Architecture
Distribution
Formats (ex: XML)
Life cycle
Dynamic Management of Business Objects
Distribution / synchronization
Save / restore (disaster recovery)
Data flow management
E.g., network capacity planning
Part
II:
Arc
hit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 13/31
Functional Architecture and Object-Oriented Design
Functional Architecture means to decompose:
Into functions and sub-functions, in a top-down manner
Descriptive normalization: (input, output, transformation, roles, …)
Functional Architecture is no longer self-contained(vs. 20 years ago)
A narrow focus on functional architecture leads to take business process
and data distribution issues into account too late.
When functional analysis is carried too deeply, one gets “”silos”
Functional top-down is poorly suited to the efficient use of off-the-shelf
software packages
Object-Oriented Design :
mixing functional & data model at the IS level
Functional Architecture feeds :
Business roadmaps & « level 0 » decomposition: descriptive analysis of
activities/jobs within the company
Service definition within SOA
Contributes to data architecture and business object model
Part
II:
Arc
hit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 14/31
Service Architecture
Service = Function + Interface + Contract
Service Architecture
Design a structure (clean up the call graph)
Provide meaning (to simplify change management and reuse)
“local” SOA = service-based architecture
Often linked to technology, such as « Web Services »
A new look for an old best practice
The goal is the system (and its architecture), services are a mean
“global” SOA = build a service catalog, whose rich structure is “an architecture”
Independent from technologies (Tuxedo, procedures, …)
Reuse of previous software component reuse theory, applied to large components which are pieces of the IS
The goal is to design a reusable service catalog (the lasting asset), architecture (organization) is a mean (and varies across time)
Part
II:
Arc
hit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 15/31
Process Architecture
Design a structure on top of temporal interactions:
Events
Chaining & dependencies => business logic
Goals => processes
Describe « fractal/recursive » structure for processes
Processes / sub-processes
Process families
Sharing resources: data, GUI, …
Roles (organizational alignment)
Business process mapping is the best alignment tool between
organization and information system
Process description -> services, functions, … (cf. previous slides)
Normalize / Standardize
Share / reuse / outsource
Best approach to integrate packaged software
Part
II:
Arc
hit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 16/31
Designing a Target Architecture
Functions Processes Data
Lexicography Métiers
Objets (ontology)
Object Lifecycle
Macro-functions Macro-processes
(draft)
Macro-processes
Exchanges - Content
Events
Services
Processes
Update protocol
Data Archi. v1 Process Archi v1 Service Archi. v1
Level 0
Catalog
External
Reference
Référence
externe
Part
II:
Arc
hit
ectu
re
External
Reference
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 17/31
Designing a Modular Architecture
Goal: minimize impact dispersion for new services
“Definition”: modularity is the correlation
« Distance in the code » and frequency of interaction
« Distance in the code » and « co-evolution »
Good practices :
Layered architecture (define abstraction levels)
Process Architecture (define a composition grammar)
Sharing/reuse & modularity go hand-in-hand : sub-process
identification
Event-Oriented Architecture
Pub/sub is still a one of the best modular patterns
Model-Driven Architecture:
careful design of « future-proof » data model
Service Architecture reduces unmanaged interactions
Reification of functional architecture
Abstraction/ encapsulation
Part
II:
Arc
hit
ectu
re
Cf. Part III
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 18/31
Part III
1. Complexity
the 21st century challenge for enterprises and their information
system
2. Enterprise Architecture
Anticipation in a complex world is not forecasting,
but building a « situation potential »
3. SOA and sustainability
repeatable long-term performance requires discipline /
governance
4. Cloud-ready architecture
Distributed, scalable, massively parallel computing resource for
the whole information system
Image blog
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 19/31
Strategic Issues : Tomorrow’s Information System
Complexity does not mean that some challenges are not clearly visible
Information Systems 2.0 – our customers …
want their voice heard: they need a feedback mechanism
collaborate with other customers or prospects
expect Information Systems to be always on, any time on any device
Information Systems need to be personalized & predictive
Social networks analytics, Bayesian learning, …
Semantic processing : making sense from customer & partner input
Time acceleration generates more data - need to master « big data »
analytical skills
Customers and end-users demand to be more autonomous
End-users have control over their own processing , write their own « apps »
IS must follow the customer wherever she goes, not the opposite !
Customer are « the architect of their own experience » → IS must become
interoperable platforms
Part
III:
Sust
ain
abilit
y &
SO
A
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 20/31
Sustainable Information Systems »
« develop today’s information services without preventing the ability
to develop those for tomorrow, through the overuse of resources, or
the production of un-manageable complexity».
Freely inspired from « sustainable » as defined by Brundtland Commission
(global) SOA is the only method for sustainable IS development
Not the only re-engineering method
(other approaches exist which reduce IS complexity),
but the best method to do so continuously, as a company-wide
practice (with all stakeholders), a long-term progressive &
measured effort, which generates its own reward
Clean-up : learn to remove/trip (classic from asset management)
Cf. Extreme programming (Agile Manifesto) :
Leveling the effort, continuous integration, favor simple designs
Continuous simplification, not a last-resort heroic attempt
Part
III:
Sust
ain
abilit
y &
SO
A
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 21/31
SOA & Governance :
Three steps:
Service Définition: build the business service catalog.
Starts like a functional analysis (from business processes) – but reusability and
composability is engineered during this step, trough the dialog between business
and IS.
Architecture de Service: Transform a list into a hierarchical and modular structure.
Classical difficulties and solutions (ex: refactoring) …
to avoid a « Web Services spaghetti».
Service Intégration : the « technical » step – often confused (SOA SOI).
Integration/middleware technology is quite mature nowadays
“Think globally, act locally” SOA starts with the periphery of the system
and ends with the core. Data architecture is critical from day one.
Beware of the “proof of concept” which is much harder to integrate than
to build
« Something you do, not something you buy » - David Linthicum
SOA Governance
its first goal is to establish the existence of shared artifacts (architecture
schemas, service catalogs, roadmaps) et everyone’s role (rights and
duties)
Part
III:
Sust
ain
abilit
y &
SO
A
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 22/31
For a company with expected growth g, the IT budget is defined as
the sum of the project budget P and the Operation budget O.
An application portfolio of size S is built, with life expectancy A
a given share (d) of the application portfolio is removed/destroyed
We suppose that the operation cost for an application that costs 1k€ is w k€
We also suppose that there is a productivity gain over the years of p%.
The usual two ratios that are of interest :
r = (P / O) = ration of build / run - usual measure R = (P / P + O)
n = Pn / P = % of the project budget spent on creating new applications
Sustainability Theorem : stable IT budget relative to company growth
n = [A (g + d) ] / (1 + (g+d) A) & r = [g + d + p +(1-d)/A] / [w (1 - d + A(g+d))]
Corollary
Easy to plug into an Excel spreadsheet
Example: w = 25%, d = 5%, p = 4%, A = 10,
we get R = 42% and n = 33%. (no surprise )
How to improve r ?
Clean-up old applications
Increase productivity for operations
Sustainability’s Equation Part
III:
Sust
ain
abilit
y &
SO
A
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 23/31
SOA as a discipline : Architecture-oriented Services
How get reusability, across the enterprise (sharing)
and across time (reuse) ?
Abstraction
Trade-off between too generic and too specific
Modularity
Relies on process and event architecture
Composability
Horizontal (Process) : Common (Pivotal) Object Model
Vertical (Functional) : Parametric Polymorphism
API Model Discipline
Manage versions !
API (meta) data model : worth some effort !
Each CIO becomes a software editor
This is more an art than a science
Part
III:
Sust
ain
abilit
y &
SO
A
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 24/31
MDA: where Abstraction and Modularity start
Data model is the key factor for modularity & flexibility
A few rules drawn from experience:
Reify links
Reify roles
Hierarchy products vs. complex ontologies
Group/individual substitution
Work on the meta-model
Think « object »
Ontology before description, encapsulation (hiding vs. transparency)
The data model defines the IS « situation potential »
Difficulty of strategic planning in a complex world
scenario practices: what if …
… we were … one of our competitor ?
… we outsourced this activity ?
… we provided this service to other companies (SaaS)
Part
III:
Sust
ain
abilit
y &
SO
A
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 25/31
Part VI
1. Complexity
the 21st century challenge for enterprises and their information
system
2. Enterprise Architecture
Anticipation in a complex world is not forecasting,
but building a « situation potential »
3. SOA and sustainability
repeatable long-term performance requires discipline /
gouvernance
4. Cloud-ready architecture
Distributed, scalable, massively parallel computing resource for
the whole information system
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 26/31
Cloud Computing : Scalable Resources & Services
Two angles :
Cloud as a computing resource : a CIO issue
Cloud as a service platform : joint ownership
A « new » kind of resource
The heir of grid and « commodity servers »
Fast provisioning
A new kind of business model
CAPEX to OPEX (rent)
Full scalable (up and down)
Part
IV:
Clo
ud-r
eady A
rchit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 27/31
The Innevibilaty of Cloud Computing
Better TCO
Commoditization
economy of scale (operations)
Utilization rate (from sharing)
PUE (Power Usage Effectiveness)
Parallel computing performance
Example from the movie industry
MapReduce & Hadoop
Distributed computing reliability
Massive redundancy …
State-of-the-art availability (99.9% to 99.99%)
Flexibility
Scalability
Pay-as-you-grow
Part
IV:
Clo
ud-r
eady A
rchit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 28/31
Don’t Confuse Clouds with Smoke
Data privacy & security
A major concern for most citizens
Not all data centers are equal in front of hackers
Data architecture must distinguished hot/warm/cold
Brewer’s Theorem (Distributed computing is still hard )
Consistency (all views obtained through different process represent
the same information)
High-availability (relies on massive replication)
Fault-tolerance (disaster recovery)
Cloud, Caching and Peer-to-Peer Architecture
« Cloud projection over the edges »
Digital homes future architecture
Part
IV:
Clo
ud-r
eady A
rchit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 29/31
SOA is not scale-free / speed of light is still a constraint
Latency is still a constraint
Tens of ms to access the cloud
Performance is not guaranteed (or price is no longer cheap)
Service « scale » (size) is crucial
SOA is not scale-free
A classic rule of « performance management » for SOA
Even more important with a cloud-architecture
RCA: Rich Client Architecture
Used for SaaS and MMORPG … for a reason
Applies to heavy transactional contexts
Part
IV:
Clo
ud-r
eady A
rchit
ectu
re
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 30/31
Cloud-ready Architecture
A cloud-ready architecture supports progressive migration
of both front-office & back-office services to the cloud
SaaS for front-office is easier : way to start …
Portal integration, then mash-up
Variation of a service-oriented
architecture
Requires to:
Leverage Web technology
“think platform” cf. “What would Google Do”
Build catalogs of
Web APIs
Part
IV:
Clo
ud-r
eady A
rchit
ectu
re
Annuaire UDDI
Applications interactives(ex: portail)
Processus
événements
Batchs
service Propriété clés:• Stateless•Gestion explicite du contexte•Contrat de service
Référentiels(données)
« Cloud »(Internet)
Applications Ressources
Infrastructure (ex: ESB asynchrone)
Infrastructure (ex: ESB synchrone)
service
Mash-ups
Yves Caseau – Complexity, Modularity, Abstraction – November, 2011 31/31
Conclusion
Models are key for:
agility
situation potential (seizing opportunities)
cost management
Innovation in the digital world happens in the source code
agile development (end user / designer / developer)
Fail often to succeed early (Eric Riess)
Lean IT: no delays, quality in, fast delivery, less code,
short intervals (small batches), customer participation,
VSM & « muda » removal
You should prepare to the advent of massively parallel computing
Cloud or multi-core
Even for traditional back-office functions
From VNG to DCG