enterprise service bus lowering the cost of integration date: 9/1/2009 dan mccreary president dan...
TRANSCRIPT
Enterprise Service BusLowering the cost of integration
Date: 9/1/2009
Dan McCrearyPresidentDan McCreary & [email protected](952) 931-9198
M
D
Metadata Solutions
Copyright 2008 Dan McCreary & Associates 2
M
D
Agenda/Objective
• What is an ESB?• What is enterprise messaging?• Calculating the ROI on enterprise service bus
Copyright 2008 Dan McCreary & Associates 3
M
D
Why Build an ESB?
• Three primary business benefits– Precision– Consistency– Agility
Precision Consistency Agility
Copyright 2008 Dan McCreary & Associates 4
M
D
Precision
• Requirements reference Business Capabilities and are defined in business approved terms and data elements.
• Business terms and data elements are stored in the metadata registry
• Metrics and business rules reference data elements that are in the registry
• Shared processes and services use data elements that are in the registry
Precision
Copyright 2008 Dan McCreary & Associates 5
M
D
Consistency
• There is a common compensation processing model and each system/product maps into that common model.
• The processing model references Thrivent Business Capabilities, business terms, data elements, rules and metrics that are in the registry.
• Everyone (architects, analysts, developers, testers) use the same definitions for business terms, data elements, rules and metrics – all stored in the registry.
Consistency
Copyright 2008 Dan McCreary & Associates 6
M
D
Agility
• A single change to elements in the registry allows all models to be updated
• A single change to models can more seamlessly be reflected in updates to applications and services
• Changing and testing the models, then transforming the models into executable code allows faster response to change, and higher quality of delivered changes.
Agility
Copyright 2008 Dan McCreary & Associates 7
M
D
Quality Metrics for an ESB
Flexibility
Consistency
Maintainability
Utility
Products
StandardizedCalculations
StandardizedInformation
Components
Interfaces
Orchestration
Rules
Plan Components
Rules and Policies
Participants
ATAMOutput
Copyright 2008 Dan McCreary & Associates 8
M
D
The Challenge
• N applications that need to communicate with each other
• 1st application needs N-1 connections
• 2nd application needs N-2 connections
• 100th system needs 99 new adapters created
• O(N*N)• 100 systems = O(10,000
interfaces)
A1
A2
A3
A4
A5A6
A7
AN
Copyright 2008 Dan McCreary & Associates 9
M
D
The Solution
• Build a central EAI information bus that has “adapters” to each of the applications
• O(N) adapters
A1
A2
A3
A4
A5A6
A7
AN
EAIBus
Copyright 2008 Dan McCreary & Associates 10
M
D
The Challenge
• How to define an Adapter for each system so that other applications can “subscribe” to its data and process services
• Make the transport reliable
Application
EAI Bus
Adapter
MessagingProtocol
Copyright 2008 Dan McCreary & Associates 11
M
D
Incoming!
XMLHas this webthing gone away yet?
web
Copyright 2008 Dan McCreary & Associates 12
M
D
Publish/Subscribe Securely
Subscriber 1
Publisher 1
Enterprise Service Bus
Publisher 2 Publisher 3
Subscriber 2 Subscriber 3
MetadataRegistry
Copyright 2008 Dan McCreary & Associates 13
M
D
3 EAI Architectures
Presentation
Database
MethodMethod
Presentation
Database
Presentation
Method
Data
Copyright 2008 Dan McCreary & Associates 14
M
D
The Heart of the Enterprise
The Metadata Registry
A metadata registry is a central location in an organization where metadata definitions are stored and maintained in a controlled method.
http://en.wikipedia.org/wiki/Metadata_registry
Copyright 2008 Dan McCreary & Associates 15
M
D
Quality
• How do you measure it?– Quality Metrics
• How can you put in “benchmarks” to make sure it is improving?
Copyright 2008 Dan McCreary & Associates 16
M
D
Reuse Rule #1
If you don’t know about it
…you can’t reuse it
• If we have any asset, how does a person know it exists?• What are the corporate repositories of reusable assts?• How easy is it to use and be notified if an asset changes?
Copyright 2008 Dan McCreary & Associates 17
M
D
Metadata Publishing Workflow Funnel
• Develop a simple workflow system for publishing data elements
• Include harvesting areas of simple glossary-of-terms found in documentation, web sites and by using metadata “scrapers” to inventory all columns in relational database systems
• Get stakeholder teams to “accept” a data elements, review them and take on the data stewardship role for these data elements
• Commit to change-control only after data elements are marked “approved for publication” by over 50% of the stewardship team
UnderReview
Approved forPublication
GlossaryOf Terms
InitialDraft
Metadata H
arvesters
Copyright 2008 Dan McCreary & Associates 18
M
D
The Garden Metaphor
• Beautiful gardens don’t just grow by themselves• They need to be designed, cultivated and
maintained• Getting out the weeds is a key part of all beautiful
gardens (removing duplicate elements)
Copyright 2008 Dan McCreary & Associates 19
M
D
Data Stewardship
• An organization needs to believe that data and the management of enterprise data is a strategic goal of the enterprise
• At it core the ongoing data stewardship process in not a function of IT, it is an enterprise value instilled in each business unit
• We need to find data steward champions in each business unit• IT can facilitate the maintenance of data but the business unit needs to
be empowered
I Love Precise Data Definitions
I Love Precise Data Definitions
Copyright 2008 Dan McCreary & Associates 20
M
D
Federated Search
• Search many sources using a single tool
Copyright 2008 Dan McCreary & Associates 21
M
D
Search ResultsInternal Data Standards External Data Standards
Copyright 2008 Dan McCreary & Associates 22
M
D
Rules
Process
Data
Rules
Can they be separated?
Copyright 2008 Dan McCreary & Associates 23
M
D
Rules and Data
Rules should always be expressed in terms that are consistent with corporate metadata standards
–Tae Sondreal, Business Rules Expert
But how do we show this in our POC?
Copyright 2008 Dan McCreary & Associates 24
M
D
Assume a Future SOA ESB
• Each system has adapters that send and receive data in standardized XML formats governed by an enterprise metadata registry
Adapter Adapter Adapter Adapter
Adapter Adapter Adapter Adapter
Enterprise Service Bus
Copyright 2008 Dan McCreary & Associates 25
M
D
What are SOA-Centric Rules?
• Rule are used by any business process that executes conditional logic on a fragment of XML data on the Enterprise Service Bus
XMLFile
Rule Rule Rule
Exception Exception Exception
Happy Path
BusinessEvent
Copyright 2008 Dan McCreary & Associates 26
M
D
Data Type Rules
A rule that only looks at a leaf on a and XML tree
• Example:– Is -1.O3 a valid number?– Is “2008-28-29” a valid date?
• Rules do not depend on any resources outside of that node
• Executes quickly using streaming memory
• “Leaf-level rules”• Not applied to branches• Use w3c XML Schema datatype
standards• Most code tables under 1,000 values• Patterns validation using regular
expressions• Enterprise SimpleTypes are stored in
the metadata registry with the data element definition
scope
See w3c XML Schema datatypes
Copyright 2008 Dan McCreary & Associates 27
M
D
Structural Rules
• Rules that express the structure of a document
• What branches are valid?• What branches are required and
how many repeating values (cardinality)?
• Can be expressed by XML Schema constraints
• Output is a valid/invalid Boolean value for the entire document
• Executed in a single pass even for gigabyte files
• No need to keep the entire document in working memory
XMLSchema.xsd
Copyright 2008 Dan McCreary & Associates 28
M
D
Transformation Rules
• Rules that change an XML document either by adding or changing structure
• Rules split up documents or combine documents
• Expressed as XQuery or XSLT
• Stored in transformation libraries that promote reuse
• Classified by the Enterprise Integration Patterns (Content Enrichment, Content Filter etc.)
A B C
C = A + B
Copyright 2008 Dan McCreary & Associates 29
M
D
Conditional Rules
• Rules that can be take every node in the tree into account
• Expressed as a Boolean (true/false) outcome
• Used to redirect a document in a workflow
• Expressed as XPath expressions using Schematron rules file and compiled into an XSLT file
• XForms application checks for valid data element definitions
A B C
If C does not equal A + B then reportan error.
Ruleregistry
Metadataregistry
Copyright 2008 Dan McCreary & Associates 30
M
D
Service Rules
• Rules that require sending a subset of the XML data to an external web service
• Depends on an enterprise resource on a remote server
• Required consideration of unreliable networks and services
WebService
database
Copyright 2008 Dan McCreary & Associates 31
M
D
Classification of Rules in the SOA Realm
• Simple Business Rules– Data type rules (XML Schema data types)– Structural rules (XML Schemas, ER relationships)– Done quickly using streaming technology (SAX)
• Medium Business Rules– Transformation and Conditional rules– Stated as XSLT or if-then-else expressions– Assume and in-memory model (DOM)
• Service Rules– Any check that requires data not easily stored in a simple XML
Schema or a rules files– Address validation– CIF lookup– Legacy services require web service adapters
Copyright 2008 Dan McCreary & Associates 32
M
D
Rules in the Registry
• We have datatype, structural and services in the registry but we do not yet have a registry of transformation and conditional rules
MetadataRegistry
Property DataElements
WantlistsXML Schema
Version Registry
SimpleTypesSubset
Generation
ServiceRegistry
Copyright 2008 Dan McCreary & Associates 33
M
D
Rule Registry
• The Rule Registry would contain rule sets• Each rule set would be controlled by a group• Rule sets contain many Rules• Each Rule contains many And (Or) expressions that are either of type assertion or
report• Expressions my only reference registered data elements or literal values
RuleRegistry
RuleSet
RuleAnd (Or)
Expression
contains many
contains many
contains many
reference many
DataElement
MetadataRegistry
registered in theGroup
Controlled by
assertions reportliteral
Copyright 2008 Dan McCreary & Associates 34
M
D
Scope of Proof-of-Concept
• We can do an architectural proof-of-concept today as long as we don’t have to deal with network reliability issues right away
• Network reliability issues need to be considered for production systems
• Can be simulated with “wrapper” web services
Copyright 2008 Dan McCreary & Associates 35
M
D
Rule as a Service
• Rule Engines can be implemented as services that hangs off of the ESB• They have XML/REST interfaces• The take XML data in and send XML data out• They can live within the eXist XQuery engine or DB2 v9 or MS-SQL 2005 or Oracle 11g• XML Schemas define their inputs and outputs
Adapter Adapter Adapter Adapter
Adapter
Rule EngineAdapter
Enterprise Service Bus
xml xml
Copyright 2008 Dan McCreary & Associates 36
M
D
Distributed Computing Issues
• What if a remote web service is down?
• Request will time out after timeout interval (by default – 15 seconds)
• Ideally we would have a “watcher” that would watch over these calls and keep trying until the remote service returns
• How can this be added to the XQuery environment?
Copyright 2008 Dan McCreary & Associates 37
M
D
Reliable Web Services Issues
• When you call any service on a remote computer it gets transparently put in a local message queue
• The XQuery waits till the result returns
• People are notified when messages are not returned with an acceptable timeframe
• Example WS-Reliable Messaging
Copyright 2008 Dan McCreary & Associates 38
M
D
What is Orchestration?
• The ability for any web service to call other web services
WebService
Call Web Service X Call Web Service Y
Call Web Service X
Request
ResponseAlso known as a “composite”Web service
Copyright 2008 Dan McCreary & Associates 39
M
D
XQuery
• XQuery is inherently a choreography language
xquery version "1.0";
let $y := doc(‘http://www.example1.com/service-y.xq’)/my-xpath-into-y
let $z := doc(‘http://www.example2.com/service-z.xq’)/my-xpath-into-z
return
<results>
{$y}
{$z}
</results>
x.xq
Copyright 2008 Dan McCreary & Associates 40
M
D
Strategy
• Continue to build an infrastructure that is consistent with the metadata registry– Selection list are context sensitive
• Use simple XML-centric standards (like XPath)– Rules expressed in terms of metadata
• Leverage “as-you-type” user interfaces (see suggest demo)
Copyright 2008 Dan McCreary & Associates 41
M
D
Thank You!
Please contact me for more information:• Native XML Databases• Metadata Management• Metadata Registries• Service Oriented Architectures• Business Intelligence and Data Warehouse• Semantic Web
Dan McCreary, PresidentDan McCreary & Associates
Metadata Strategy [email protected]
(952) 931-9198