xa transaction & integration use case with openesb
TRANSCRIPT
XA transaction & integrationUse case with OpenESB
Agenda
Pymma Consulting: Services versioning with WSDL 2
XA for an ideal worldXA and the real worldXA and IntegrationOpenESB featuresConclusions
THE IDEAL WORLD
Pymma Consulting: Services versioning with WSDL 3
The ideal world
Pymma Consulting: Services versioning with WSDL 4
Process
Context N
Context N
Partner 1
Context N
Partner 2
Context N
Partner 3
Before the process starts
The ideal world
Pymma Consulting: Services versioning with WSDL 5
Process
Context N+5
Context N+3
Partner 1
Context N+2
Partner 2
Context N +7
Partner 3
During the process
The ideal world
Pymma Consulting: Services versioning with WSDL 6
Process
Context N+8
Context N+5
Partner 1
Context N+6
Partner 2
Context N +9
Partner 3
An error occurs during the process
The ideal world
Pymma Consulting: Services versioning with WSDL 7
Process
Context N+8
Context N+5
Partner 1
Context N+6
Partner 2
Context N +9
Partner 3
Partners are informed about the error
The ideal world
Pymma Consulting: Services versioning with WSDL 8
Process
Context N
Context N
Partner 1
Context N
Partner 2
Context N
Partner 3
Partners and process rollback to the original contexts
The ideal world
Pymma Consulting: Services versioning with WSDL 9
Process
Context N
Context N
Partner 1
Context N
Partner 2
Context N
Partner 3
Using XA for transaction and recovery
XA
XACommitRollback
CommitRollback
CommitRollback
XA
The ideal worldXA is perfect for our ideal world We are all familiar with XA or two phases commit transaction– Mainframe, Database, Transaction manager, J2EEserver
XA Transactions provide – Once and only once delivery /consistency
XA (2 phases commit) has been the basis of 70’s 80’s and 90’s IT softwares
Pymma Consulting: Services versioning with WSDL 10
THE REAL WORLD
Pymma Consulting: Services versioning with WSDL 11
The ideal worldXA is a concept for an care bear’s worldThe ideal world is difficult to implementXA needs special conditions for working well– Uniformity / Proximity
Uniformity – Does not work in a non uniform environment– Not implemented with heterogeneous applications
Proximity– XA works only when partners collocated – Being on the same network is not a collocation
In the 00’s, Internet blasted XA and the ideal world
Pymma Consulting: Services versioning with WSDL 12
Internet blasted the ideal world
From stand alone to multi-partners
Pymma Consulting: Services versioning with WSDL 13
Silo Application
Module
Module
Module
Collaborate
Providerclient
Partner
Partner
Client
Provider Partner
Partner
Client
Provider
Internet blasted the ideal world
From one to multiple technologies
Pymma Consulting: Services versioning with WSDL 14
IBM Mainframe
Unix
Mainframe
Linux
WindowsWeb
Software PackageExternal
Partners
FTP
Http
JMS
NFS
TCP/IP
Java
.Net
C++
Cobol
Internet blasted the ideal world
Web applications need more power than ever
Pymma Consulting: Services versioning with WSDL 15
Up to 10k users
Up to 100 M users
Internet blasted the ideal world
Web applications need more power than ever
Pymma Consulting: Services versioning with WSDL 16
Server
ServerServer
ServerServer
ServerServer
ServerServer
ServerServer
The Real worldIn the 00’s – Many projects failed in distributed environment– Business & management worried about this fact
• Why IT could not provide what it did in the 70’s 80’ and 90’s ?
High management said:– Difficulties to implement XA came from
• A lack of knowledge from IT people• Bad technologies and platforms choices
Moreover in the 00’s big editors lied– Saying their tools support distributed XA– Helped by famous analysts
Let’s explain why XA was so difficult to implementPymma Consulting: Services versioning with WSDL 17
The real world
Pymma Consulting: Services versioning with WSDL 18
Client /Server Architecture
SQL
18
Centralised Architecture
The real world
The network is not a simple extension – Latency creates partitioning– One group can work an not the other
Pymma Consulting: Services versioning with WSDL 19
Latency
M a n y a r c h i t e c t s t h o u g h t t h a t d i s t r i b u t e d a r c h i t e c t u r e s b e h a v e l i k e c e n t r a l i z e d a r c h i t e c t u r e s .
The real world
You create partitions – When you work with many platforms – When you use different technologies – When you create a geographical distribution
Partition means:– A part of your system can work when one is stopped
Pymma Consulting: Services versioning with WSDL 20
Partition
The real world and CAP theorem
Pymma Consulting: Services versioning with WSDL 21
In 2002 the CAP theorem saved us– CAP stands for • Consistency Availability and Partitioning
– Set out by Eric Brewer • Mathematically proved by S. Gilbert et N. Lynch (MIT)
CAP is a mathematical answer to – The 00’s management and Editors
Let’s see what it says
Pymma Consulting: Services versioning with WSDL 22
The real world and CAP theorem
CAP Theorem (Consistency, Availability, Partition)– We need 3 properties needed for a network: consistency, availability
and partitioning. At most, it is possible to get two of them.
C A
P
• Consistency + Availability • Single site / Cluster database• Two phase commit
C AP
• Consistency + Partition • Distributed database / Distributed looking• Pessimistic looking
C AP
• Availability + Partition • Distributed cache/ DNS• Optimistic looking
The real world and CAP theorem
Pymma Consulting: Services versioning with WSDL 23
office
officeoffice
The real world and CAP theorem
First Case I want consistency
Pymma Consulting: Services versioning with WSDL 24
Office Brussels
Office Berlin
Office Nice
Inform
Inform
Lock
C AP
The real world and CAP theorem
First Case I want consistency
Pymma Consulting: Services versioning with WSDL 25
Office Brussels
Office Berlin
Office Nice
Get Data
Data not available
C AP
Data not available
The real world and CAP theorem
First Case, we chose consistency
Pymma Consulting: Services versioning with WSDL 26
Office Brussels
Office Berlin
Office Nice
Inform
Inform
Unlock
C AP
Data are consistent but not always available
The real world and CAP theorem
Second case I want availability
Pymma Consulting: Services versioning with WSDL 27
Office Brussels
Office Berlin
Office Nice
Get data
C AP
The real world and CAP theorem
Second case I want availability
Pymma Consulting: Services versioning with WSDL 28
Office Brussels
Office Berlin
Office Nice
process data
C AP
Get data
process data
The real world and CAP theorem
Second case I want availability
Pymma Consulting: Services versioning with WSDL 29
Office Brussels
Office Berlin
Office Nice
C AP
Put dataPut data
Data are availablebut not always consistence
Pymma Consulting: Services versioning with WSDL 30
The real world and CAP theorem
CAP Theorem (Consequences)– In a partitioned environment
• We cannot get at the same time consistency and availability• Mathematically demonstrated that XA is not available• We must choose which property to promote
– Ex : statistics data/ Historic data / transfer / Customer account
Why is the CAP theorem is new for us? – Limited latency in the local area network– Database editor propaganda – Not teach in the universities yet – Many architects ignore CAP theorem existence – Similitude with the uncertainly principle of Heisenberg
Pymma Consulting: Services versioning with WSDL 31
The real world and CAP theorem
Concept difficult to admit: Impact on mentality and habit– Business people are use to design with the concept of
transaction– B.As write specifications relying on XA transactions – Going out transactions seems a regression for the business– Most of architects are comfortably relying on transactions
to provide consistency– They don’t know how to do without it– Some developers cannot live without “Commit and
Rollback”
The Real world (summary)In the 00’s IT world changed– Internet destroyed the ideal world
It introduce biodiversity in IT systems– Many partners (some not XA)– Multiple technologies (some not XA)– Many protocols (some not XA)
XA does not work in a non uniform environmentXA doesn’t exist in the real worldXA doesn’t exist in real life
Pymma Consulting: Services versioning with WSDL 32
INTEGRATION AND XA
Pymma Consulting: Services versioning with WSDL 33
Integration and XA
Pymma Consulting: Services versioning with WSDL 34
All together Manual Validation
Receive Transfer order by email
Check rights
Swift transfer
Send email for confirmation
Check account
Banker validation
Log transaction
Pymma Consulting: Services versioning with WSDL 35
Integration and XA
Integration means– Multiple partners– Multiple Technologies– Multiple protocols
Typically an heterogeneous environment– Impossible to apply XA
For these environment could we provide:– Guarantee of delivery – Consistency
Question: – Can integration stand with strong consistency and high
guarantee of delivery?
OPENESB FEATURES: CONSISTENCY & GUARANTEE OF DELIVERY
Pymma Consulting: Services versioning with WSDL 36
OpenESB features
OpenESB is an integration tool– Relying on JBI specifications (Java Business Integration)
OpenESB proposes natively advanced features – High availability – Quality of services – Consistency
We will review these featuresThey are used by architects & designers to increase quality
Pymma Consulting: Services versioning with WSDL 37
Pymma notation
Pymma Consulting: Services versioning with WSDL 38
BindingComponent
Binding Component XA implementation
Service without side effects
Service without side effects
XA Implementation
Service with side effects
Service with side effects
XA Implementation
Pymma notation
Pymma Consulting: Services versioning with WSDL 39
BPEL
Catch
Compensation
The process is atomic
The process is persistent
Regular process
When an error occurs
Compensation part of the
process
N
RedeliveryN times
Integration and business processes
Pymma Consulting: Services versioning with WSDL 40
Short processesFew partnersAutomatic tasks
Emitter
Receiver
Integration process
BPEL
Few seconds
max
Automatic tasks
Integration process
OpenESB Technical concepts
Pymma Consulting: Services versioning with WSDL 41
Longer processesMany partnersAutomatic & Manual tasks
Emitter
Receiver
Business process
BPEL
Fromsecondsto weeks
Automatic tasks
Manual tasks
Business process
Service acknowledgement
Pymma Consulting: Services versioning with WSDL 42
partner partner
JBI environment
Acknowledgement
Acknowledgment is required in a JBI environment
Emitter Receiver
Acknowledgement with BPEL
ACKBPEL
ACK
BPEL Atomic property
Pymma Consulting: Services versioning with WSDL 43
Emitter
Receiver
ACK is sent just after the reception: Atomic = false
ACK
BPEL
Receive
Activity
Activity
invoke
BPEL Atomic property
Pymma Consulting: Services versioning with WSDL 44
Emitter
Receiver
No ACK is sent when a non catch error occurs in the process : atomic = true
ACK
BPEL
Receive
Activity
Activity
invoke
XX
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 45
Emitter
Receiver
process
Simple process
A
B
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 46
Emitter
Receiver
process
Simple process with side effect
Side effect
A
B
Catch
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 47
Emitter
Receiver
process
Simple transactional process with side effect
A
B
There is no RollbackThere is no Rollback
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 48
Compensation
A
Side effects
C: AService A
Compensation
Increases side effects
Suppresses or decreasesside effects
Service A
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 49
Insert in database
Delete in database
Compensationcomplete
Write in audit file
Write in audit file
Compensationpartial
Put message in Topic
Put message in Topic
CompensationPartial
Add number in a total
Subtract number in
total
Compensationcomplete
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 50
CService A
CService B
CService C
CService D
Service A Service B Service C Service D
Normal process
Compensation process
CService A
CService B
CService C
CService D
Service A Service B Service C Service D
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 51
Save a record
C: delete the record
DB
Insert into ....Audit service.
V1
Delete from ....SQL
Save a record
C: delete the record
Write in file
File
Audit service. V2
Delete from ....SQL
Side effect and compensation
Pymma Consulting: Services versioning with WSDL 52
Save a record
C: delete the record
DB
Insert into ....
Audit serviceV1
Delete from ....
Save a record
C: delete the record
Write in file
Audit serviceV2
Delete in file
File
F-1(x)
F-1(x)
F(x)
F(x)
Qualities & Redelivery Extension
Pymma Consulting: Services versioning with WSDL 53
Redelivery Extension– Max attempts N times– Wait time – On failure • Delete / error / redirect / suspend
BPEL
BPEL
N
N
N
N
N
NBPEL
Qualities & Redelivery Extension
Pymma Consulting: Services versioning with WSDL 54
BPEL Http
Non reliable connection3
Wait 2s
External Service
BPEL JMS MQ2
Redirect
Http Web Service
2
Error
FTP FTP ServerSuspend
BPEL2
Monitoring
Try to fix the problem
Persistence Principle
Pymma Consulting: Services versioning with WSDL 55
C C CC
Side effect
No transactional process
After a failure
Full transactional process
RollbackAfter a failure
Qualities & Redelivery Extension
Pymma Consulting: Services versioning with WSDL 56
C C CC
BPEL internal context
DB
C C CC
BPEL internal context
DB
Pymma Consulting: Services versioning with WSDL 57
Qualities & Redelivery Extension
Limits side effect Increases guaranty of deliveryDecreases the performances– Powerful DB server can be used– Cache distributed instead DB
Solutions can be found to use efficiently persistence
C C CC
BPEL internal context
DB
High availability
Pymma Consulting: Services versioning with WSDL 58
Machine 3
Machine 2
Machine 1
Domain
Cluster
Cluster
Instance
Instance
ESB
ESB
Instance
Instance
ESB
ESB
Instance
Instance
ESB
ESB
App.
App.
App.
App.
App.
App.
App.
App.
Node agent
Node agent
Node agent
Cluster
High Availability
OpenESB has been designed for multi-instance– Native design – Don’t use containers cluster features
It provides easily high Availability Architects uses this features– To increase Quality of services– Useful to secure processes and services
Must be associated with good BP designs
Pymma Consulting: Services versioning with WSDL 59
HOW TO USE OPENESB FEATURES
Pymma Consulting: Services versioning with WSDL 60
Cleverness and pragmatismThere are 3 delivery types– 0,n / 1,n / 1,1 delivery
There is not system that can really provide 1,1– XA system are close from it – OpenESB can provide 1,n in a non XA environment– Many native OpenESB features to achieve 1,n delivery
Pragmatism – Do not try to reach 1,1– increase business and management awareness about it
Cleverness– Identify what is really important, exceptional, vital– Don’t focus on the exceptional– Don’t forget that human is the best to process exception
Pymma Consulting: Services versioning with WSDL 61
Cleverness and pragmatismCleverness start with BP design– Avoid stateful processes promote stateless
• Use hydration and dehydration
– Avoid Multipart conversation – Plan entry features to avoid doubloon– Plan compensation messages at the design level
Be pragmatic– Cancelation message is often better that transaction– Human is very smart to solve complex issue
• Recording vital data in a data is critical• Process it in a simple way• Process automatically the most common errors• Human or external process to solve complex error
Pymma Consulting: Services versioning with WSDL 62
80% -20%
Cleverness and pragmatism
Spend time on happy ways and simple errors– Design and processing are straightforward– Simple error can be process easily
• QOS / Redelivery • Compensation • Atomicity
– That’s architect and designer job
Don’t spend time on exceptional– Don’t try to replace (or replay) XA– What happens if a meteorite fails on my computer– You will increase development and budget– increase maintenance complexity
Pymma Consulting: Services versioning with WSDL 63
Cleverness and pragmatism
Does this philosophy works– Google, Ebay, Amazon follows this philosophy– They cancelled XA for scalability or Consistency
Pymma helps its customers to follow this way – Large to very large projects > many K men days– KBC Bank, Leroy Merlin, Kiabi, IFDS– They all need reliability and consistency – But scalability and efficiency for integration
They found in OpenESB a solution We are sure you will find in OpenESB your solution
Pymma Consulting: Services versioning with WSDL 64
QUESTIONS AND COMMENTS
Pymma Consulting: Services versioning with WSDL 65
THANK YOU