ulf fildebrandt · about the presenter • ulf fildebrandt • studied computer science • works...
TRANSCRIPT
Software modular bauen
Ulf Fildebrandt
Some words before
This presentation should…
• …give some insights in big software products
• …and how the structure of these products is managed
About the presenter
• Ulf Fildebrandt
• Studied computer science
• Works for SAP since 1998
• Starting with C++, once SAP decided to use Java part of the Java stack
• Product architect of SAP Netweaver Composition Environment (Java stack)
• Program architect of SAP Netweaver 7.30
• Program architect of Integration on Java on demand stack
Agenda
• Problem statement
• Goals
• Means to explain
• Basics of modularity with OSGi
• Principles and patterns
– Design Patterns (GoF)
– SOLID
• System architecture (for decoupling)
• Comparison of frameworks
Problem statement
Findbugs Analysis by Structure 101
SAP Netweaver Structure
„In every phenomenon the beginning remains always the most notable moment.“
Thomas Carlyle
Findbugs Analysis
• Famous analysis done by Ian Sutton
• Can be found at http://structure101.com/blog/2008/11/software-erosion-findbugs/
• Uses Structure 101 (see also JAX 2012 innovation award for Restructure 101)
Findbugs 0.7.2
Findbugs 0.8.6
Findbugs 0.8.7
Findbugs 1.0.0
Findbugs 1.3.0
Findbugs 1.3.5
Server Analysis
• Own analysis tool of SAP Java server
• Structure of current Netweaver server is shown
• Not only Java dependencies (packages)difference to Structure 101
– Usage Types
– Software Components
– Packages
Structure of Complex System – IOverview of product
http://pwdfm295.wdf.sap.corp:9090/makeresults/ce.lc.reporting/NW730EXT_SP_COR/gen/dbg/java/packaged/full/_doc_pkg/content/index.html
Structure of a Complex System – IIBasic Scenario
Structure of a Complex System – IIIDetailed dependencies
Goals
Substitutability
Extensibility
„...it is almost always incorrect to begin the decomposition of a system into modules on the basis of a flowchart. We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others.“
David Parnas
Extensibility
Module
B Module
C
Module
A Module
D
Module
E
Module
F
A module is a self-contained component of a system, often interchangeable, which has a well-defined interface to the other components.
http://en.wiktionary.org/wiki/module
Substitutability
Module
B Module
C
Module
A Module
D
Module
E
Module D‘
A module is a self-contained component of a system, often interchangeable, which has a well-defined interface to the other components.
http://en.wiktionary.org/wiki/module
Module
F
Goal: Sustainable Software
Module
B Module
C
Module
A Module
D
Module
E
System can evolve over time and modules can be replaced without impacting the other modules.
Means to explain
Levels of Architecture
Web Application as an example
„Programming without an overall architecture or design in mind is like exploring a cave with only a flashlight: You don't know where you've been, you don't know where you're going, and you don't know quite where you are.“
Danny Thorpe
Levels of Architecture
Coding-Architektur
Komponenten-architektur
Systemarchitektur
Same principles can be applied on all levels of architecture
FacadeFactory
Dependency Injection
Liskov
Web Application
Demo
Basic modularity with OSGi
Packages
Services
Lifecycle
Modularity Maturity Model
„Good fences make good neighbors.“
Robert Frost
(Internal Packages) Bundle
Package Structure of a Bundle
ExportedPackage
Exported Package
ImportedPackage
ImportedPackage
• Bundle exports only defined packages
• Packages can be imported from other archives (bundles)
• All other packages are only visible inside the bundle
Package Dependencies of a Bundle
Bundle
ExportedPackage
Exported Package
Bundle
ImportedPackage
ImportedPackage
Bundle
Service Structure of a Bundle
Exposed Service
Exposed Service
Used Service
Used Service
• A service is an implementation of a Java interface
• Bundle exposes services to service registry
• Used services are taken from service registry
Bundle
Service Dependencies of a Bundle
Exposed Service
Exposed Service
Used Service
Used Service
Bundle
Exposed Service
Bundle
Exposed Service
• Service implementations are taken from service registry (not shown here)
Lifecycle of a Bundle
installed
resolved
uninstalled
starting
active
stopping
start
stop
install
resolve
uninstall
uninstall
How to achieve modularity...
• OSGi is a framework on top of Java
• Complements Java very good
• ...but modularity does not come for free
Modularity Maturity Model – I• Level 1: Ad Hoc
At the Ad-hoc level, there isn't any formal modularity. A flat class path issued with a bunch of classes with no, or limited, structure.They may use 'library JARs' to access functionality but typically results in a monolithic application.
BenefitsThe main benefit of this is the low cost and low barrier to entry.
• Level 2: ModulesModules are explicitly versioned and have formal module identities, instead of merely classes (or JARs of classes). In particular, and a key point of this, is that dependencies are done against the module identity (including version) rather than the units themselves.Maven, Ivy, RPM and OSGi are all examples of where dependencies are managed at the versioned identity level instead of at the JAR level.
Benefits– Decouple module from artefact
– Clearer view of module assembly
– Enables version awareness through build, development and operations
– Enables module categories
http://wiki.osgi.org/wiki/Modularity_Maturity_Model
Modularity Maturity Model – II• Level 3: Modularity
Modules are declared via module contracts, not via artefacts. The private parts of the modules are an implementation detail.In this level, dependency resolution comes first and module identity is of lesser importance.
Benefits– Fine-grained impact awareness (for bug fixes, implementation or client breaking changes)
– System structure awareness
– Client/provider independence
– Requirement-based dependency checking
• Level 4: Loose couplingThere is a separation of interface from implementation; they are not acquired via factories or use constructors to access the implementations. This provides a services-based module collaboration (seen in OSGi, but also present in some other frameworks like Spring or JNDI to hide the construction of the services from the user of those services).In addition, the dependencies must be semantically versioned.
Benefits– Implements a client/provider independence
http://wiki.osgi.org/wiki/Modularity_Maturity_Model
Modularity Maturity Model – III• Level 5: Devolution
At the devolved level, artefact ownerships are devolved to modularity-aware repositories. They may support collaboration or governance for accessing the assets by relation to the services and capabilities required.
Benefits– Greater awareness of existing modules
– Reduced duplication and increased quality
– Collaboration and empowerment
– Quality and operational control
• Level 6: DynamismProvides a dynamic module life-cycle, which allows modules to participate in the life cycle events (or initiate them). Will have operational support for module addition/removal/replacement.
Benefits– No brittle ordering dependencies
– Ability to dynamically update
– Can allow fixes to be hot-deployed and to extend capabilities without needing to restart the system
http://wiki.osgi.org/wiki/Modularity_Maturity_Model
Principles and patterns- Design Pattern (GoF)
Facade
Factory
„Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.“
Christopher Alexander
Factory – I
• A factory externalizes the creation of objects
• Creation and usage of objects is decoupled
• Pattern definition works for coding With factory:
IDataAggregator aggregator = DataAggregatorFactory.getInstance();
Without factory:DataAggregator da = new DataAggregator();
A factory is an object for creating other objects. It is an abstraction of a constructor. http://en.wikipedia.org/wiki/Factory_(software_concep
t)
Factory – IIApply on component level
• Externalization of creation of objects is a general pattern
How can OSGi help:
• Service registry provides an abstraction between provider and consumer of a service (object instance)
• Creation of instances of a service can be handled by frameworks:
– Declarative services
– Blueprint
Facade – IA facade is an object that provides a simplified interface to a larger body of code, such as a class library.
http://en.wikipedia.org/wiki/Facade_patternFacade:
public interface IDataAggregator {public List<IDataItem> get();
}
Implementation of facade:final class IdentityDataAggregator extends DataAggregator {
@Overridepublic ArrayList<IDataItem> get() {
List<IDataItem> itemList = new ArrayList<IDataItem>(); ...
return itemList; }}
Facade – IIApply on component level
• Access to a component as a general pattern
How can OSGi help:
1. Services are Java interfaces
1. Service implementations are accessed via interface = facade
2. Exported packages are the only external visible entities of a bundle
1. Not exported packages are not accessible
2. Clear definition of a facade of the bundle
Principles and patterns- SOLID
Dependency Injection (SOLID)
Liskov (SOLID)
„Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.“
Christopher Alexander
Dependency Injection – IDependency injection involves at least three elements:
• a dependent consumer,• a declaration of a component's dependencies, defined as interface
contracts,• an injector (sometimes referred to as a provider or container) that
creates instances of classes that implement a given dependency interface on request.
The dependent object describes what software component it depends on to do its work. The injector decides what concrete classes satisfy the requirements of the dependent object, and provides them to the dependent.
http://en.wikipedia.org/wiki/Dependency_injection
Dependency Injection – II
prop
erty
m
etho
d
Class 1 Class 2
met
hod
prop
erty
m
etho
d
met
hod
prop
erty
m
etho
d
met
hod
prop
erty
m
etho
d
met
hod
Dependency Injection – IIIApply on component level
• Reverse the dependency is a general pattern
How can OSGi help:
• Service registry provides an abstraction between provider and consumer of a service (object instance)
• Injecting dependencies can be handled by additional frameworks
– Declarative services
– Blueprint
Web Application
Demo – Declarative
Services
LiskovLet q(x) be a property provable about objects x of type T. Then q(y) should be provable for objects y of type S where S is a subtype of T.
http://en.wikipedia.org/wiki/Liskov_substitution_principleFacade = T:
public interface IDataAggregator {public List<IDataItem> get();
}
Implementation of facade = S:final class IdentityDataAggregator extends DataAggregator {
@Overridepublic ArrayList<IDataItem> get() {
List<IDataItem> itemList = new ArrayList<IDataItem>(); ...
return itemList; }}
Example to apply coding patterns
© SAP AG 2009. All rights reserved. / Page 46
Definition of a Composite Application
Wikipedia: A Composite Application is an application built by combining multiple existing functions into a new application.
Extended Definition: Composite Applications are user centric applications supporting highly collaborative and dynamic business processes which span beyond functional, system, and organizational boundaries by using data and functions provided as services by platforms and applications.
© SAP AG 2009. All rights reserved. / Page 47
Anatomy of Composite Applications
Business EntitiesLocal Remote
CO
MP
OS
ITE
AP
PL
ICA
TIO
N
CRM BW ERP Systems
BA
CK
EN
D
Workcenter
Composite Process
Role 1 Role 2
Step 1 Step 2 Step 3 Step 4
Service EnablementServicesServices Services
Enterprise Service Bus(optional)
Remote Services
LocalServices
Business Entities,Services
UI UI UI User Interfaces
DEMO Composite Application Structure
Simple approach: build a framework on top of the tools, but…
© SAP 2007 / Page 49
Problem of an Integration Framework
Domain
Domain Tool
Domain
Domain Tool
Domain
Domain Tool
Integration Framework
R R R
Domain
Domain Tool
New toolsThe integration framework does not have a reference to the new tool
Installation problem„All or nothing“-approach, because all tools have to be installed
Note: same problem appears by integrating all domains with a common layer at the bottom, e.g. modeling framework/repository
Basic principle: Adapter mediates between tool and capability
© SAP 2007 / Page 50
Adapter-based Integration
Adapter connects tool and capabilities Tool does not reference the capability
functionality integration is completely done by the adapter
Adapter is completely separated from tool tool is built on minimal requirementstool is able to run without integration
Target is exposing interfaces and extension points (Eclipse)
Source is implementing extensions and using public APIs
Source Target
Adapter
© SAP 2007 / Page 51
Adapter Architecture for CapabilitiesInteraction between Domain and Capability
Domain
Integration Framework
Adapter
Domain Tool
R
Capability
R
Domain
Adapter
Domain Tool
R
R
Domain
Adapter
Domain Tool
R
R
Consistent visualization of a Composite Application Graphical overview and relations Display consistency or inconsistencies between parts of a Composite Application
Integration Framework (Composite Designer) Pluggable infrastructure to integrate separated tools Optional extensions to allow step-wise integration
Tools can start with a basic integration and extend it later
Integration framework provides extension points
Tool provides extension points
Facade
[Overview][display main entities in one diagram]
© SAP 2008 / Page 52
[Overview][display main entities in one diagram]
Public interface IElement
Base protocol for all entities that Composite Designer can deal with.
Elements can appear in two basic forms:
Resolved: such elements point to an entity from a concrete domain(e.g. J2EE, Web Dynpro). They usually appear in the form of an graphical element.
Unresolved: such elements cannot be resolved in the adapter which created them, but usually in a different adapter. For example the Process Composer adapter could create an unresolved element for a Web Dynpro Component by just specifying its key, and only the Web Dynpro adapter is able to resolve it into a complete (resolved) element.
© SAP 2008 / Page 53
Main element factory provider registration<extension point="com.sap.ide.comp.pers.core.modeler.graphicalElementFactories " id="wd" name="WD Content in Overview"> <factory dcType="Web Dynpro" class = "com.sap.ide.comp.pers.adapter.wd.WDGraphicalElementFactory" layer="User Interface"/></extension>
Signature of: Interface IGraphicalElementFactory
java.util.List<IResolvedElement> getMainElements(IGraphicsContext context)
Factory
[Relations][display dependencies between main elements]
© SAP 2008 / Page 54
[Relations][display dependencies between main elements]
Public interface IElement
Base protocol for all entities that Composite Designer can deal with.
Elements can appear in two basic forms:
Resolved: such elements point to an entity from a concrete domain (e.g. J2EE, Web Dynpro). They usually appear in the form of an graphical element.
Unresolved: such elements cannot be resolved in the adapter which created them, but usually in a different adapter. For example the Process Composer adapter could create an unresolved element for a Web Dynpro Component by just specifying its key, and only the Web Dynpro adapter is able to resolve it into a complete (resolved) element.
© SAP 2008 / Page 55
Dependency contributor registration<extension point="com.sap.ide.comp.pers.api.dependencyContributors"> <contributor
class="com.sap.ide.comp.pers.adapter.wd.WDDependencyContributor"/></extension>
Signature of: Interface IDependencyContributor
java.util.List<IResolvedElement> getIncomingElements(IResolvedElement element) java.util.List<IElement> getOutgoingElements(IResolvedElement element) IResolvedElement resolveElement(IResolutionContext context)
Separated interface
System Architecture- for decoupling
Architecture Layer
SEDA
„Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.“
Christopher Alexander
Architecture Layer – I
System
Platform
System Layer
…
Sub system …
Module
Package
…
…
• A platform (in terms of runtime framework) hosts a system and is a versioned artifact
• A system consists of a set of sub-systems and is a versioned artifact running on a platform
• A system layer groups sub-systems providing structure and direction in terms of dependencies and is a layer which has no associated versioned artifact
• A system layer incarnation is the bill of materials or assembled sub-systems of a system layer relevant for a specific use case
• A sub-system consists of a set of modules and is a versioned artifact
• A module contains packages and is a versioned artifact
• A package contains components– A component is a File– A file is of type resource or source
Architecture Layer (as is) – II
System Layer
Package analysis of web application
Uses Structure 101 (see also JAX 2012 innovation award for Restructure 101)
Module = Subsystem = Bundle
Pac
kag
e
Architecture Layer (to be) – III
Data Source Interface
Data Aggregator Interface
Data Provider
Aggregator
UIData Display
(Servlet)
Data Source 1 Data Source 2
DataAggregator 1
DataAggregator 2
Data Aggregator Interface
Data Source Interface
Architecture Layer – IV ConQAT (to-be diagram)
• ConQAT (https://www.conqat.org)
• Compared to Structure 101: extensible because of open source
– Used to implement modularity metrics
• Easy integration in build process (Maven) automatic check of to-be and as-is
Modularity Metrics – ICoupling: determines the coupling of this component to other components and is computed based on instability (Robert C. Martin).
– I = Ce / (Ca+Ce) 1..0 (Ce (efferent) = outgoing, Ca (afferent) = ingoing)
– Consequence: the more components THIS component depends upon, the more instable it is (0= stable, 1=instable)
(Internal Packages) Bundle
ExportedPackage
Exported Package
ImportedPackage
ImportedPackage
Ca = usages of exported packages
Ce = imported classes in packages
Modularity Metrics – IIRelational Cohesion: a component should consist of cohesive elements, otherwise, it should be splitted. Average number of internal relationships per type.
– rc = Td / T (Td.. type relationships that are internal to this component; T.. number of types within the component)As classes inside an assembly should be strongly related, the cohesion should be high. On the other hand, too high values may indicate over-coupling. A good range for RelationalCohesion is 1.5 to 4.0. Assemblies where RelationalCohesion < 1.5 or RelationalCohesion > 4.0 might be problematic. Therefore rcI is the normalized RC, having value 1 for 1.5 to 4, decreasing to 0 outside this range based on gaussian bell curve.
Modularity Metrics – IIIEncapsulation: Components should encapsulate knowledge and offer a slim interface.
– ep = pt / T (pt = private Types, T = all types) pt = types in internal packages, T = types in internal packages + types in exported packages
The Overall Modularity Score is now defined as:
– M = ((1-I) + rcI + ep) / 3.0
(Internal Packages) Bundle
ExportedPackage
Exported Package
ImportedPackage
ImportedPackage
Demo ConQAT – I
Demo – web
application
Demo ConQAT – II
Demo – server from
SAP
Don‘ts in Layering – I
Schicht
SchichtClass 1
Parameter Class
Class 2ruft per Reflection auf
ruft auf
verwendet
Don‘ts in Layering – II
Schicht
SchichtClass 1 Eclass 1
Class 2
referenziertruft auf
Eclass 2
String
String
referenziert
SEDA – Functional modularization
Stage
Thread
Thread
Thread
.
.
.
Queue
Stage
Thread
Thread
Thread
.
.
.
Queue
Stage
Thread
Thread
Thread
.
.
.
Queue
Old Rendering@Overrideprotected void doGet(HttpServletRequest req, HttpServletResponse resp) throws
ServletException, IOException {StringBuilder buffer = new StringBuilder(200);PrintWriter writer = resp.getWriter();
renderHeader(buffer);
List<List<String>> list1 = dataAggregateService.getData(req.getParameter("type"));
TableObject analyzedTable =new TableObject(list1, DisplayServlet.ANALYZED_DATA, buffer);
renderTable(analyzedTable);
List<List<String>> list2 = dataAggregateService.getData("identity");
TableObject originalTable =new TableObject(list2, ORIGINAL_DATA, buffer);
renderTable(originalTable);
renderFooter(buffer);writer.write(buffer.toString());
}
Old Rendering@Overrideprotected void doGet(HttpServletRequest req, HttpServletResponse resp) throws
ServletException, IOException {StringBuilder buffer = new StringBuilder(200);PrintWriter writer = resp.getWriter();
renderHeader(buffer);
List<List<String>> list1 = dataAggregateService.getData(req.getParameter("type"));
TableObject analyzedTable =new TableObject(list1, DisplayServlet.ANALYZED_DATA, buffer);
renderTable(analyzedTable);
List<List<String>> list2 = dataAggregateService.getData("identity");
TableObject originalTable =new TableObject(list2, ORIGINAL_DATA, buffer);
renderTable(originalTable);
renderFooter(buffer);writer.write(buffer.toString());
}
Rendering Decoupled@Overrideprotected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {StringBuilder buffer = new StringBuilder(200);PrintWriter writer = resp.getWriter();
renderHeader(buffer);
t.setDataAggregateService(dataAggregateService);
triggerRendering(req.getParameter("type"),
DisplayServlet.ANALYZED_DATA);triggerRendering("identity", DisplayServlet.ORIGINAL_DATA);
String message1 = getResult();StringTokenizer tokenizer1 = new StringTokenizer(message1, "\n");String type1 = tokenizer1.nextToken();String body1 = tokenizer1.nextToken();
String message2 = getResult();…
Trigger Renderingprivate void triggerRendering(String type, String header) {
Connection connection = null;// Create the connection.try {
ActiveMQConnectionFactory connectionFactory =new
ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER,ActiveMQConnection.DEFAULT_PASSWORD,ActiveMQConnection.DEFAULT_BROKER_URL);
connection = connectionFactory.createConnection();connection.start();
// Create the sessionSession session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);destination = session.createQueue(in);
// Create the producer.MessageProducer producer = session.createProducer(destination);producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
TextMessage message = session.createTextMessage(type + "\n" + header);
producer.send(message);…
Get ResultActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL);connection = connectionFactory.createConnection();
connection.start();
// Create the sessionSession session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);Queue destination = session.createQueue(out); // Create the consumerMessageConsumer consumer = session.createConsumer(destination); Message message = consumer.receive(); if ( message instanceof TextMessage ) {
TextMessage textmessage = (TextMessage)message;text = textmessage.getText();
}return text;
Dispatcher Threadwhile(true) {
Queue destination = session.createQueue(in);MessageConsumer consumer = session.createConsumer(destination);Message message = consumer.receive();…
Queue destination = session.createQueue(out);MessageProducer producer = session.createProducer(destination);TextMessage message = session.createTextMessage(type + "\n" + body);producer.send(message);…
}
Comparison of frameworksExample: OSGi vs. Maven + Spring
Criteria for Comparison
• Modules: software is structured in clearly separated units.
• Decoupling: software modules have clearly defined interfaces and frameworks support development based on interfaces.
• Repository: software modules are decoupled entities that can be stored in a repository.
• Versioning: software modules are versioned and framework can handle multiple versions.
OSGi vs. Maven + CDI
OSGi is not the only technology, other technologies serve the same purpose
Versioning
Module
Decoupling
Repository 0
2
4
OSGi
Versioning
Module
Decoupling
Repository 0
2
4
Maven
CDI
Summary
Key take-aways
• Big software systems have to be structured, but more high-level than packages
• Additional frameworks support modularity (e.g. OSGi, Maven, Spring, CDI implementations…)
• Modularity can be measured by metrics