placement of bpm runtime components in an soa environment
TRANSCRIPT
© 2012 IBM Corporation
IBM Software Group Placement of BPM runtime components in an SOA environment
Presenter: Kim Clark Authors: Kim Clark ([email protected]) Brian Petrini ([email protected]) Version 1.6
IBM Software Group
© 2012 IBM Corporation
Related article
Please note, the following article has been published that covers the concepts in the first half of this presentation: “The placement of BPM runtime components in an SOA” http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html
25/01/14
IBM Software Group
© 2012 IBM Corporation
How complicated can a simple sequence of boxes be?
Process
IBM Software Group
© 2012 IBM Corporation
Common questions arising when considering BPM and SOA
https://collaboration.opengroup.org/projects/soa-ref-arch
• Should all requests go through business process management (BPM)? • Is BPM above or below the enterprise service bus (ESB)? • Are processes also services? • Should BPM handle page navigation? • Should BPM perform integration?
IBM Software Group
© 2012 IBM Corporation
Consumers
Are business processes just another type of consumer?
Services (Exposure)
Operational Systems (Applications & Data)
Process Runtime (BPM as a consumer)
Other consumer
Service Providers Service C
onsumers
Other consumer
Could/should we remove the business process layer altogether? Removes temptation to assert that all requests should go through business process layer.
IBM Software Group
© 2012 IBM Corporation
Sequences of steps Where could they occur / Where should they occur?
Service Exposure
Operational Systems (Applications & Data)
Integration
Consumers
Business Process
IBM Software Group
© 2012 IBM Corporation
People based processes
Service Exposure
Operational Systems (Applications & Data)
Integration
Consumers
Business Process
IBM Software Group
© 2012 IBM Corporation
What makes a business process suitable for BPM?
Performing the process provides value to the business The process contains individually business relevant steps Business relevant data flows through the process The process follows a relatively structured path The steps within the process are performed by multiple roles/teams. The process changes over time as a result of changes in the business
Create Account
Capture Details
Send Documents
Sign Documents
Activate Account
Customer
Call centre
Back office
Notify Customer
New revenue!
Existing customer
New customer
-- - --- -- --- -- -- --- -- ------ -- --- -- -
-- - --- -- --- -- -- --- -- ------ -- --- -- -
-- - --- -- --- -- -- --- -- ------ -- --- -- -
-- - --- -- --- -- -- --- -- ------ -- --- -- - -- - --- -- --- -- -- --- -- ------ -- --- -- -
-- - --- -- --- -- -- --- -- ------ -- --- -- -
IBM Software Group
© 2012 IBM Corporation
Consumers
BPM as a consumer of services
Services (Exposure)
Operational Systems (Applications & Data)
Business Process (Orchestration)
Process Runtime (as a consumer)
Service Providers Service C
onsumers
A business process orchestrates (consumes) services in order to automate steps in the process
Human task
System task
Exposed service
Back end system
Misleading representation
of an interaction
Interaction
Key
IBM Software Group
© 2012 IBM Corporation
Consumers
Positioning of the integrated BPM UI
Services (Exposure)
Operational Systems (Applications & Data)
Business Process (Orchestration)
Process Runtime (consumer)
Service Providers Service C
onsumers
Typically an internal BPM UI is provided to enable users to view and work their tasks.
Integrated BPM UI
IBM Software Group
© 2012 IBM Corporation
Consumers
Introducing an external BPM user interface
Services (Exposure)
Operational Systems (Applications & Data)
Business Process (Orchestration)
Process Runtime (consumer)
Integrated BPM UI
Service Providers Service C
onsumers
Often some, or all of the human tasks need to be served up by an external custom portal due to the specific requirements of the graphical user interface.
retrieveTask() updateTask() completeTask()
Independent BPM UI
IBM Software Group
© 2012 IBM Corporation
Systems acting as both providers and consumers simultaneously
§ The layered architecture is a logical model, not a physical deployment
§ Components may appear more than once if they perform different roles
§ Many operational systems are both providers and consumers
Services (Exposure)
Operational Systems (Applications & Data)
System A (as a provider)
Service Providers Service C
onsumers
Consumers System B (as a consumer)
System A (as a consumer)
System B (as a provider)
IBM Software Group
© 2012 IBM Corporation
Consumers
The process runtime acting as a provider of services
Services (Exposure)
Operational Systems (Applications & Data)
Business Process (Orchestration)
Process Runtime (as a consumer)
Integrated BPM UI
Process Runtime (as a provider)
Independent BPM UI
Service Providers Service C
onsumers
In reality, the business process runtime also becomes a service provider in this situation providing services for external systems to read and manipulate tasks. So the BPM component now lives in two places on the diagram, each with a specific role. BPM is both above and below the service layer
retrieveTask() updateTask() completeTask()
Inte
rnal
inte
ract
ion
with
in
busi
ness
pro
cess
runt
ime
API
IBM Software Group
© 2012 IBM Corporation
Pros and Cons of fully decoupling the process API Pros: • Enables hiding of routing, versioning, security models etc. • Becomes a standardize process capability, independent of technology and
implementation • Standardized monitoring and controlling service level agreements, throttling
traffic • Standardized governance: Enforcing policies for things such as encryption,
deprecation. • Optionally enables standardization of data model, but see Cons below. Cons: • Significant up front work before the first project can use the API • Results in another layer to be maintained, more skillsets involved operationally. • May render the standard API product documentation less effective, and new
documentation may need to be created and maintained. Particularly true if a new data model is introduced.
Recommendations • Only consider decoupling where multiple highly independent consumers are
present, each with strong reliance on well defined service level agreements. • Focus on requirements around traffic management, and security. • Avoid replacing/translating the APIs data model. This is a lot of work both up
front and ongoing for very little gain.
IBM Software Group
© 2012 IBM Corporation
Page navigation
Service Exposure
Operational Systems (Applications & Data)
Integration
Consumers
Business Process
IBM Software Group
© 2012 IBM Corporation
Separating Process, Page Navigation and Page Design
Loan Requester
Loan Approver
System
Approve Loan
Process Loan
Capture Loan Details
Capture Personal Details
Capture Financial Details
First name
Second name
Date of birth
Personal details
etc.
What’s wrong with this process model?
Multiple steps in the same human swimlane of a BPMN diagram suggest there is an issue with the granularity of the process modeling. It implies they could have been performed by the same person in one go, so they are not really separate steps. We should not model this low level page navigation in the BPMN diagram.
IBM Software Group
© 2012 IBM Corporation
Separating Process, Page Navigation and Page Design
Loan Requester
Loan Approver
Capture Loan Application
System
Approve Loan
Process Loan
Process
Capture Loan Application
Loan Details
Personal Details
Financial Details
First name
Second name
Date of birth
Personal details
etc.
Page navigation
Page
Page navigation should be modeled separately from the BPMN swimlane diagram.
IBM Software Group
© 2012 IBM Corporation
Separation of concerns when creating a custom UI for BPM
Independent user interfaces for BPM should also take on the page navigation. Example issues • Page navigation depends on the
page data • There are often data dependencies
across pages • Pages/data may need to be pre-
loaded based on navigation • The ideal UI data model will be
different to the process model
Process Layer
Presentation Layer
Page navigation
Page
Process
IBM Software Group
© 2012 IBM Corporation
Consumers
Is a process also a service?
Services (Exposure)
Operational Systems (Applications & Data)
Business Process (Orchestration)
Process Runtime (consumer)
Integrated BPM UI
Service Providers Service C
onsumers
Process Runtime (provider)
submitLoanApplication()
Service Components
startProcess()
User Interface
Inte
rnal
inte
ract
ion
with
in
busi
ness
pro
cess
eng
ine
API
“Is a process a service?” – No! But a service can result in the initiation of a process.
IBM Software Group
© 2012 IBM Corporation
Processes with integration
Service Exposure
Operational Systems (Applications & Data)
Integration
Consumers
Business Process
IBM Software Group
© 2012 IBM Corporation
Single Transaction for all invocations
Multiple Transactions across invocations
Human Tasks
Terminology: Orchestration vs. Process vs. Composition? Orchestration is a sequence of “invocations” of systems, or humans
Appear to the business as a single step.
Do not include human interaction from the business.
Take long enough that the business need to be aware of their intermediate in-progress states.
Include human interaction, or interaction with slow responding systems.
Orchestration
Composition Process
Note: These are not industry standard terms, but derive from common usage observed in the field
IBM Software Group
© 2012 IBM Corporation
Core types of Orchestration: Process vs. Composition
Process Makes calls to mature high level services Often triggered (i.e. one way call) rather than invoked as a two way call Where it is invoked as a two way interaction, the caller is typically asynchronous (i.e. not a user) and therefore the service level agreement is throughput based rather than response time based Stateful persistence of the steps in the process Events can correlate with the running process Often involves human interaction to perform some tasks within the process
Composition Grouping of relatively fine grained interactions Typically called in real-time in a request/response (two way) interaction style. Response time is the primary driver for the service level agreement Common for aggregation functions Some or all the granular interactions may not themselves be exposed as re-usable services Generally state free Never involves human interaction during the composition
IBM Software Group
© 2012 IBM Corporation
Separating the business process from low level composition
Loan Requester
Loan Approver
System
Approve Loan
Save Loan Details
Capture Loan Application
What’s wrong with this process model?
Multiple steps in the same system swimlane of a BPMN diagram suggest there is an issue with the granularity of the process modeling. It is likely we are modeling the separate systems users have to update today. In the future process this multiple update will be automated, so it is no longer be relevant to model the individual steps on the business process diagram. We should not model this low level integration logic in BPMN diagram.
Save Personal Details
Save Financial Details
IBM Software Group
© 2012 IBM Corporation
Pros/cons of moving composition out of the process layer
Orchestration modelled and implemented within the process layer. Good for visibility, but process much more complex. Granular services exposed for re-use. Interactions traverse many layers (poor performance)
Orchestration modelled and implemented in the integration layer. Hides integration complexity from process. Course grained services exposed for re-use Interactions are close to systems (good performance)
Service Exposure
Operational Systems (Applications & Data)
Integration Layer
Business Process Layer
Composition
Business Process
a b c
Service Exposure
Operational Systems (Applications & Data)
Integration Layer
Business Process Layer
Business Process
b a c
IBM Software Group
© 2012 IBM Corporation
Just a selection of the different patterns of orchestration
Integrated workflow
Stateful integration
Aggregation Isolated
transaction
Composition
Exceptions only
Process
Stateless* engine Stateful* engine
* This refers to persistence of orchestration state
Real time retrieval of data from
multiple systems
Real time updates to multiple systems
that can be combined into a single update
Straight through processing across
systems that can’t be combined
transactionally
People based exception handling
Processes integrating people
and systems
IBM Software Group
© 2012 IBM Corporation
“Process” patterns
Integrated workflow
Stateful integration
Exceptions only
Process
Stateful engine
IBM Software Group
© 2012 IBM Corporation
Pure workflow process People based process with no system integration
All data is stored in the process. State is stored as each user works a task Model changes can be in more in the hands of the users. Examples • Paper based processes • Procedures involving multiple teams of people Primary benefits: • Reduction/removal of paper • Improved work distribution/prioritisation • Management visibility on process status • Common understanding of process model Implementation: Stateful orchestration engine with good task user interface capabilities.
System based activity
People-based activity
Queued events
State persistence
IBM Software Group
© 2012 IBM Corporation
Integrated workflow process People and systems are involved in the process
Enables progressive automated of tasks Users directly involved in modelling of process. IT required for integration. Examples • Processes where elements of data are duplicated across multiple systems • Processes where system updates are unnecessarily complex for users Benefits: • Benefits of Pure Workflow • Reduce/re-assign workforce • Improve end-to-end time • Improve data consistency Implementation: Stateful orchestration engine with good integration capabilities. Significantly simplified if implemented in conjunction with a service oriented architecture.
System based activity
People-based activity
Queued events
State persistence
IBM Software Group
© 2012 IBM Corporation
Process with people for exceptions only Main path is straight through, people handle known exception paths
Users can progressively refine exception path rules to increase automation. A step in the direction of Straight Through Processing (STP) Examples • Approval processes rules enable some auto-approval Benefits: • Benefits of Integrated Workflow Process • Manage proportionally higher volumes Implementation: Orchestration engine with integration and rules capabilities.
System based activity
People-based activity
Queued events
80%
20%
State persistence
IBM Software Group
© 2012 IBM Corporation
“Composition” patterns
Stateful integration
Aggregation Isolated
transaction
Composition
Stateless engine
IBM Software Group
© 2012 IBM Corporation
Aggregation No transactions required, just gathering data
Data gathered from multiple sources and merged into a single result. By far the most common requirement. Most solutions need convenient access to rich data. Examples • Gathering quotations from multiple insurance brokers (parallel) • Combining customer data from multiple systems (sequenced) Benefits • Relatively simple implementation since no data integrity/transactionality requirements. • Failures part way through can be simply re-tried • Hides complexities of data translation/merge and connectivity to disparate systems Implementation: Ideally performed as close to the back end systems as possible. Integration engine preferable due to strong data merging capabilities, and more direct connectivity to back end systems. Challenges • Should you wait for all responses • How do you merge parallel requests • Which layer in the architecture should this be performed.
System based activity
IBM Software Group
© 2012 IBM Corporation
Isolated transaction Multiple transactional calls to the same system
Provides a real-time response regarding the status of the update to the caller. Commonly used to commit changes requested by user interfaces providing immediate feedback of success. Examples • Updates to multiple tables in the same database. e.g to create an order and its order items. Benefits • Ensures data integrity across multiple calls to the same system • Failures will be fully rolled back, so requests can be simply re-tried. Implementation: Ideally done as close to the data as possible for performance, and simplicity of the transactional protocols. Highest performance will be within the back end system (e.g. a stored procedure). Next best is a integration engine that can handle the transactional protocol (such as JDBC). Challenges • Transactions typically hold resources (e.g. threads/memory) whilst in progress • The aggregate of all the steps may take longer than the transaction timeout. • Transactions containing multiple steps can cause deadlocks if coding guidelines are not adhered to.
Transaction Boundary
System based activity
Single phase commit
IBM Software Group
© 2012 IBM Corporation
Distributed transaction Data integrity critical across multiple systems
Requires a independent transaction controller to perform a global transaction across more than one system. This requires that the systems are enabled for two phase commit transactions. Examples • Committing changes to multiple databases • Committing changes to a queue and a database. In reality two phase commit is more commonly used “within” a system, rather than across multiple systems. For the reasons noted in “Challenges” Benefits • Ensures data integrity across calls multiple systems • Failures will be fully rolled back, so requests can be simply re-tried.
Challenges • Few system owners will allow a two phase commit to be performed against their systems by external parties due concerns over resource locking. • Two phase commit require the maintaining of a transaction log so systems can re-align should a failure happen mid transaction. This adds interdependencies that complicate design of high availability and disaster recovery. • Deadlocks are much more difficult to diagnose than with single phase commit. • Two phase commit is much more heavyweight in CPU both for the end systems, and for the caller than single phase commit.
Transaction Boundary
System based activity
Two phase commit
IBM Software Group
© 2012 IBM Corporation
Enriched update Only the final update matters, everything else is preparation
Transaction Boundary
System based activity
read non-critical update
critical update
*Definition: “Critical Update” Any request that performs a change to data (create, update or delete) that MUST be resolved in the case of a failure, either with a transactional rollback, or with compensatory actions if the update has already been committed.
A complex scenario is simplified to focus on the one key update, thereby removing the need for intermediate state. Examples • Checking for existence of a parent objects before creating a child object – e.g. checking customer exists before creating order. • Validating data before performing update – e.g. checking for stock inventory before submitting an order. Benefits • Enables a complex composition yet still removes the need for stateful orchestration. Challenges • Re-tries result in repeating all of the pre-critical steps • Invocations deemed to be “non-critcial” may result in confusion during diagnosis of issues.
IBM Software Group
© 2012 IBM Corporation
Event distribution/broadcast We only need to know it will be done, not that it has been done
Messages are placed in a number of queues within a single transaction. Examples • Useful for updates to systems that are not expected to be instantaneously up to date. E.g. data warehouses, audit logs • Useful for notification type requests (SMS, email etc.) Benefits • Provides a swift answer to the caller • With persistent queues, the messages should never be lost, and will be applied to their destinations eventually. Challenges • Destination systems have not received the data when the response is given • The time taken for destinations to receive the update is not in the control of the composition • Should problems occur processing the messages, the composition, and therefore also the caller are unaware. Therefore reliant on external exception handling.
Transaction Boundary
System based activity
Queued events
IBM Software Group
© 2012 IBM Corporation
Basic Interaction Types
Provider Requester getCustomer(CustomerID)
returns Customer
Provider Requester submitOrder(Order)
a) Fire and forget
b) Request-response
A messaging transport is used. The request is “one-way” sending data to the provider, but with no response. The requester need only wait for the message to be received by the transport – i.e. it is non-blocking. This is the style typically used to initiate a process. i.e. we do not wait for the process to complete.
A synchronous transport is used. The requester requires response data, and must wait for the provider to respond. The provider must be available at the time of the request and must process the request immediately. This is the style typically used with compositions. i.e. the caller requires a response from the composition.
IBM Software Group
© 2012 IBM Corporation
Stateful Integration – The pattern with two homes
Integrated workflow Aggregation Isolated
transaction
Composition
Exceptions only
Process
Stateless engine Stateful engine
Stateful integration
IBM Software Group
© 2012 IBM Corporation
Stateful Integration Process Zero or near zero human interaction
All steps are system invocations. Multiple separate transactional interactions are required. State must be stored in between each step, ideally as part of the same transaction. Examples • Data synchronisation. e.g. “Change of Address” • “Straight Through Processing” (STP). e.g. payments processing Benefits • Enables orders of magnitude higher volumes. e.g. moving sales to the internet • Enables precise data integrity across disparate systems Implementation: Depends significantly on non-functional requirements – throughput/volumes. See next slide for options.
System based activity
Queued events
State persistence
Transaction Boundary
IBM Software Group
© 2012 IBM Corporation
Implementation options for Stateful Integration Process
Orchestrated interactions Pros Mature products available (e.g. BPM) First class notion of “process” Modelled process visible in implementation Monitoring/reporting at process level Process version migration support Clearer modelling of exception paths Easier to introduce human interaction Cons Higher CPU/network due to persistence Solution will be split between business process integration layers
Chained interactions Pros Higher performance due to minimal persistence Can be implemented entirely in the integration layer Cons Largely a custom solution No notion of “process”, only the individual steps Process “model” not visible in implementation Custom work for effective monitoring/reporting Exception paths harder to understand Custom work to migrate to new processes Custom work to introduce human interaction
Queued events State persistence
IBM Software Group
© 2012 IBM Corporation
What orchestration patterns are required?
Service Exposure
Operational Systems (Applications & Data)
Integration
Consumers
Business Process Integrated workflow
Page Navigation
Enriched Update
Aggregation Stateful integration
IBM Software Group
© 2012 IBM Corporation
Core characteristics for assessing orchestration requirements
Time-span Granularity
Human interaction
Principal objects
Application integration
Complexity
Monitoring
Flow ownership
Dynamicity Transactionality
Performance
Volumes
Business value
State management
Security
Infrastructure
IBM Software Group
© 2012 IBM Corporation
Many patterns, many possible implementation options
Integration flow pub/
sub
Business process
short running
Business process briefly
persisted
Integration flow 1:n
Integration flow 1:1
with translation
Business Process
long running system centric
Business Process
long running human centric
Integration flow 1:1
routing only
Many attempts at decision trees for implementation choices fail because they try to cover the whole many-to-many mapping from all possible requirements to all possible implementations. This is typically an impossibly hard decision tree to draw, and inconceivably complex to navigate.
Implementation Options
Requirements
more… more…
IBM Software Group
© 2012 IBM Corporation
Narrowing the complexity using a two stage decision tree
Patterns
Implementation Types
Use architectural principles to determine the relevant implementation choices for that pattern
Use requirements to choose the pattern
IBM Software Group
© 2012 IBM Corporation
Types of orchestration in relation to transactionality Do any of the requests perform critical updates*?
No
Aggregation Enriched update
Yes
Can all critical updates* be performed in one transaction?
Distributed Transaction
Stateful Integration
Yes No
How many critical updates*?
=1 >1
*Definition: “Critical Update” Any request that performs a change to data (create, update or delete) that MUST be resolved in the case of a failure, either with a transactional rollback, or with compensatory actions if the update has already been committed.
Non-transactional Isolated transaction Global transaction Multi-transaction
IBM Software Group
© 2012 IBM Corporation
No single answer Process solutions are always a combination of orchestration patterns
Service Exposure
Operational Systems (Applications & Data)
Integration
Consumers
Business Process
Data synchronization
Microflow
In-app. workflow
Screenflow/ page navigation
Stored procedure
Straight-through processing
Business process
IBM Software Group
© 2012 IBM Corporation
References
The placement of BPM runtime components in an SOA http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html
Process implementation types: Patterns based design for process-based solutions http://www.ibm.com/developerworks/websphere/library/techarticles/1004_clark/1004_clark.html Process-oriented modeling for SOA http://www.ibm.com/developerworks/library/ar-procmod1