Choosing an applicationintegration strategy
Anko DuizerClass-A
Overview
• Why integration?• What is integration?• Challenges• Common scenario’s• Which technologies does Microsoft offer?• Messaging• Design considerations• A different viewpoint: The Ring(s)• The Matrix
Why?
• Enterprises are typically comprised of hundreds applications– Custom built, acquired by a third party, legacy
systems
• Reasons– Creating a single, big application to run a
complete business is impossible– Flexibility to select “the best”– Use a collaborative partner model
Those who can not remember the past are condemned to repeat it!
The current situation
FB
E
A
D
C
What?
• Connecting– Computer systems– Companies– People
Application integration is the secure and orchestrated sharing of processes and/ or
data between applications
Integration scenarios
• Information portals
• Data replication
• Shared business functions
• Service-oriented architectures
• Distributed business processes
• Business-to-business integration
Basic challenges
• Networks are unreliable
• Networks are slow
• Any two applications are different
• Change is inevitable
• Organizational problems
Integration approaches
• Simple association
• Central hub
• The big database approach
• Message bus
• Service integration
• Hybrid
Simple association
UI
BO
BO
Data
Application
UI
BO
BO
Data
Application
Endpoint Endpoint
Central hub
UI
BO
BO
Data
Application
UI
BO
BO
Data
Application
Endpoint Endpoint
ERP System
The big database approach
UI
BO
BO
Data
Application
UI
BO
BO
Application
Message bus
UI
BO
BO
Data
Application
UI
BO
BO
Data
Application
Endpoint Endpoint
Message bus
Service integration (1)
UI
BO
BO
Data
Application
UI
BO
BO
Data
Application
Endpoint Endpoint
Service
Service integration (2)
Application
Endpoint
Service
Service Service Service
Endpoint
Application
Endpoint
Application
Endpoint
Application
Microsoft technology
• BizTalk Server 2004
• Web Services
• MSMQ
• DTS
• Indigo (not available yet)
• SQL Service Broker (not available yet)
BizTalk Server 2004
actionorchestrationdesigner
biztalk editor
biztalkmapper
messagingmanagement
development operation management
administration
tracking
messaging services
orchestration services
config
tracking
port
pipelinedesigner
applications
monitoring
Businessrules
Web Services
Host
Component
Client
Application
Host
Page
Browser
XML over HTTP
HTTP request
.NETCOMEJBPerl…
listenerproxy
proxy
Internet
MSMQ
Queue
msgmsgmsgmsg
Sender Receiver
DTS
OLE DB ODBCFixed fieldASCII delimited
Source
OLE DB ODBCFixed fieldASCII delimitedRepl. publication
Destination
DTS Data Pump
In Out
ActiveX ScriptCopyTrim String…Custom
Transforms
Indigo
Port
Service
Service
Message
Channel
Channel
Service
ServiceChannel
SQL service brokerSQL Server object DescriptionMessage Type Defines valid messages for exchange between services
Contract Specifies type of messages and their direction in a conversation, initiator or target
Queue Stores messages before sending and after receipt as result set
Service Program The part of a service broker application that reads messages from a queue and processes them.
Service Addressable endpoint for service communication
msgmsgmsgmsg
Service
Contract Serviceprogram
Queue
Message type
Messaging
• Loosely coupled
• Communicating asynchronously
• Reliable
• Makes the messaging responsible for transferring data from one application to another
Sync or async?
time
Process A
Process B
blocked
time
Process A
Process B
Basic messaging concepts
ApplicationApplication Channel
Endpoint
Endpoint
Message Translation
Routing
Messaging patternsY
X
Y
X
1)
• Fire and Forget
• No Response required
2)
• Request / Reply
• System makes request
and gets response
3)
• Needs ability to cache response from multiple requests to respond to one message
Y
XEAIEAI
EAI
B
X
4A/B)
• Multiple Applications needed to satisfy request
• 4A: XREF Look-up
• 4B: XREF Update
A C
XREF
EAI
A
X
5)
• Publish Subscribe
• A, B and C subscribe to message
B
CEAI
6A)
• Needs data from Z to be
able to update Y
Y
X EAI
Z
The message is the message!
Main design considerations
• Timing/ Latency
• Identification– Industry keys, Internal keys, Chained keys &
Matching algorithms
• Sharing entities, activities, processes and utilities
• Designing endpoints
• Message transformation
Main design considerations
• Routing
• Security
• System management
• ACID Transactions
• Service-oriented architectures
Shared data ownership
E-Commerce
Inventory
E-Commerce
Choosing
To EAI or to ETL?Data (ETL)
Data Transformation Services
Messages (EAI)
BizTalk Server 2004
Unit of modeling
“Package” – an XML definition of “workflow” and data flow
Schema that capture structure/
Contracts that capture behavior
Unit of interaction
“Synchronous” DTS pipeline for data movement
Asynchronous message passing
Instancing DTS Service controls instances Weak/opaque URI-based references
Extensibility Scripts, SQL, custom components
Intermediaries and message routing
Platform Databases Multiple
Evaluation criteriaCriterion Description
Latency How quickly is the data to be transferred?
Transformation Complexity of the transformation
Scope Application, organization or partner?
Volume Quantity of exchange by a transfer
Intrusion Degree of change to existing applications in order to effect transfer
Effort Effort required to build and maintain the solution
Organization Which solution fits best with the organization?
The Ring(s)
1. Application
2. Enterprise
3. Collaborative partners
4. Everybody else
The Matrix
BizTalk SSB Indigo MSMQ DTS
Latency Near real time
Near real time
Real time Near real time
Batch
Transformation High High N/A N/A High
Scope Enterprise & partners
App. App. App. App. & Enterprise
Volume Medium Medium Low Medium High
Intrusion Medium Low High High Low
Effort High Medium High Medium Medium
Organization tbd tbd tbd tbd tbd
The Matrix - ReloadedIndigo MSMQ SSB
Environment Any WS-* compliant
Windows SQL Server 2005 (both)
Application Any distributed application
Asynchronous Database application
Message store
In-memory or database
NT File system
SQL Server 2005
Type of message
Persistent & Non-peristent
Reliable, Express, Transaction
Transactional only
Protocol Various Various TCP only
The Matrix - Revolutions
Ring 0 1 2 3Communication Synchronous Asynchronous Asynchronous
Focus Performance Security
Integration
Security
Integration
Firewall
Indigo Yes Maybe No
BizTalk Server No Yes Yes
WS-* Maybe Yes Yes
MSMQ No Yes No
SSB Yes No No
Conclusions
• Integration is hard!
• Integration requires good thinking!
• Different technologies available, use them smartly!
• Start thinking asynchronously and message based!
References
• Books– Enterprise integration Patterns, Hohpe &
Wools– Enterprise Integration Solutions, Spackman &
Speaker
• Web– http://www.class-a.nl– http://www.objectwatch.com– http://msdn.microsoft.com/architecture
Questions ?
The Rings(s)
1. Synchronous, .NET or J2EE? , Performance, Indigo, SSB, Web Services
2. Asynchronous, MSMQ or MQSeries?, DTS, Security, WS-*
3. Asynchronous, BizTalk Server, Security, WS-*
4. Firewalls, for the rest you don’t care!