wso2 integration platform deep dive eu con 2016

WSO2 Integration Platform Deep Dive

Chanaka Fernando - Senior Technical Lead

Ramith Jayasinghe - Senior Technical Lead

Agenda

• Concepts in Integration• Demonstration• Product roadmaps/features• Q&A

Evolution of Integration

Point to Point Integration● Custom code, data formats for

majority of integration ( data extraction, business rule processing, data loading)

● Costly over time (maintenance, complexity)

● Tight Coupling

● Scalability is an issue

● Less reusability

Evolution of Integration

Use of Enterprise Service Bus ● Consistency: Clear set of guidelines (EIPs) results in repeatable and successful integration efforts/projects.

● Reusability: Tested and preconfigured components provides repeatable approach to integration.(transports/connectors/mediators etc)

● Effectiveness: A common integration framework (with related abstract concepts) improves integration developer's productivity.

Connected Business

An internally and externally connected business.

Image courtesy hRp://jdamico.net/wp-content/uploads/2010/04/collaborate-with-B2B-channel-partners.jpg

ESB were used to connect internal systems in the past.

Current need is to connect internal/external parties quickly and flexible manner.

E.g. Employees, Internal/External systems, Partners, Customers.

Connected Business

Integration is a key factor for having frictionless interactions, inside and outside the organization.

Integration and APIs - The Close Cousins

• APIs cannot replace Integration

– Integration of internal services, systems, data and cloud apis

• Cannot mangle SOA for API Management needs

• Using SOA and API in combination is the key to success of

Connected Business.

Integration and APIs - The Close Cousins

WSO2 Integration + API Management Platform

In practise API management and integration co-exists providing flexible, faster ways to connect required parties such as users, internal/external systems, employees etc.

WSO2 Integration Platform

Fundamentals of WSO2 ESB

Message Processing units- Sequences, mediators

Message entry points- Proxy Services, APIs, Inbound

endpoints.Message exit points

- Endpoints

Mediators

Example MediatorsName Description

Log Mediator Logs full or part of the message, at various severity levels ( Trace, Debug, etc)

Sequence Mediator Invokes existing sequence - Sequence name can be static or dynamic

Send Mediator Sends a message out, using static information or endpoint definition.

Datamapper Mediator Transform data from one form to another

Switch Mediator Evaluates messages contents against regular expression and invokes the corresponding mediator (switch-case-default)

Validate Mediator Validates message or parts of message against XML schema (schema can be local or in registry)

PayloadFactory Mediator Create a message payload from an existing message or from scratch

Fault Mediator Transforms current message into custom Fault message

Mediators

• Transformation Mediators• Filters, Switches, Iterate, Aggregate, Clone• Property Mediator• Script Mediators• Custom Mediators

Connectors

• A connector is a ready made and convenient

tool to reach publicly available web API’s.

• ‘Cloud to Cloud’ and ‘Cloud to Enterprise’

Integration

• ESB Connector Store : https://store.wso2.com/

Connectors

• Every connector is self-contained and independent from ESB code

• Dynamically plug in to ESB• Dynamic Tooling support with WSO2 Developer

Studio• You can write, ‘your own connector’ and just plug it

in

Sequences





- Endpoints

Proxy Service

• Acts like a virtual service.

• Receives messages and mediates

them before sending them to the

endpoint (usually an actual

service)

• Mainly to expose as SOAP Service

API/HTTP Service

• APIs can accept REST messages which allow clients

to provide additional information on how to

mediate the message

• Defined under a URL Context (e.g: /customer)

• Can define multiple resources under a URL context

• Resources

– Component of API accessed through HTTP call

– Similar to proxy service (in, out, and fault

sequences)

– Can restrict resource’s scope using URL

patterns and URI templates

API/HTTP Service

• URL mapping

– Path mappings (eg: /test/*, /foo/bar/*)

– Extension mappings (eg: *.jsp, *.do)

– Exact mappings (eg: /test, /test/foo)

• URI template

– /order/{orderId} would process /order/A0001

– /dictionary/{char}/{word} would process /dictionary/c/cat

– Use get-property to retrieve exact values

<log level="custom">

<property name="Character" expression="get-property('uri.var.char')"/>

<property name="Word" expression="get-property('uri.var.word')"/>

</log>

Inbound Endpoint

• Creating inbound messaging channels

dynamically.

– Listening Inbound – HTTP, HL7, TCP

– Polling Inbound – JMS, File/VFS

• Injects the inbound messages into an ESB message flow.

Polling Inbound endpoint in a Cluster

One Consumer Multiple Consumers





- Endpoints

Endpoints

• Logical representation of one or more real backend

service, jms queue etc.

• Address, HTTP Endpoints – A logical endpoint for an

existing BE service

• Load balancing / Failover – Group of endpoints for

existing BE services

Message Store and Message Processor

• Asynchronous/One-way Messaging, JMS

• Store and forward

• Guaranteed Delivery and Rate matching

• JMS Message broker as the persistence store – WSO2 MB, Apache

Active MQ etc.

Message Processor in a Cluster

• One consumer per cluster – Guaranteed in-order delivery

• Multiple consumers – Guaranteed delivery

One Consumer Multiple Consumers

Tasks

• Allow configuration of scheduled jobs to execute internal/external

commands

• Inject a message into a proxy service or a named sequence

Service Chaining

• Orchestrating/Combining multiple Services

Load Balancing

Enterprise Integration Patterns

https://docs.wso2.com/display/IntegrationPatterns/Enterprise+Integration+Patterns+with+WSO2+ESB

Tooling Support for WSO2 ESB

WSO2 ESB 5.0 – Upcoming release

• Beta version is available (Released on May 2016)

• Improved usability of the ESB through debugging, data mapping capabilities through developer studio tooling

• The product is released as a tuple of runtime, analytics and tooling

• Dedicated analytics distribution(DAS) for ESB specific monitoring with pre built dashboards

WSO2 ESB 5.0 – Statistics/Tracing

• Coarse grained statistics about proxy/sequence/api/endpoint


• Fine grained statistics/tracing on mediators (message level)


• Fine grained tracing on messages

WSO2 ESB 5.0 – Visual Data Mapper

• Visual Data mapper– Different input/output options (XML, JSON, Files, Schemas)

– Apply functions while mapping the data (Split, Aggregate, Uppercase, Lowercase)

WSO2 ESB 5.0 – Mediation debugger

• Debug mediation flows from the Developer studio

WSO2 ESB – What’s new in 5.0

Other new features

• JMS 2.0 support

• Websocket support

• JMS distributed(XA) transaction support

• Rabbitmq/MQTT performance improvements

WSO2 Integration Cloud (Upcoming)

WSO2 Integration Cloud (Upcoming)

Integration Platform as a Service (iPaaS)

• Running integrations on the cloud

• Analytics integration

• Develop integrations using WSO2 Developer studio

Business Processes

WSO2 Business Process Server

• BPMN 2.0• WS-BPEL 2.0• WS-HumanTask 1.1• BPEL4People

● Define and execute business processes

● Define workflows interacting with People

● Graphical process modeling/administration/monitoring

Human Task Support

WSO2 Business Rules Server

● Integrates JSR94 compliant rule engines.

● Embed drools engine.

Fundamentals of WSO2 Message Broker

Messaging Models● RPC Style

- Request/Response (there is always a response)- Synchronous (client waits for response)- Non-persistent (message is lost if something failed)

● Point to Point (Queues)- A message is delivered only once to a single Broadcast a

message to all the subscribers● Publish/Subscribe (Topics)

- Broadcast a message to all the subscribers


Messaging Models

● Publish/Subscribe (Topics)- Broadcast a message to all the subscribers

● Durable Subscriptions- Same as topic subscriptions- Subscriber offline - Messages get queued- Subscriber back online - Messages get delivered- Unsubscription needed to remove queuing


● Transactional- Send/Receive multiple messages in a transactions.- ( send/consume -> rollback, reject, commit)

● Guaranteed Delivery/Acknowledgments , QOS ( MQTT, JMS).● Persistence

- Messages kept in memory or in reliable storage.

Fundamentals of WSO2 Message Broker● Dead letter channel

- Broker keeps invalid/rejected messages in this channel.- Users can inspect/reroute/delete these messages.

● Hierarchical topics

- Map relationships between data being published onto different topics.

- Can Subscribe to part of the hierarchy.

Games.* - receives messages published to ‘Games.FootBall’

and ‘Games.Cricket’

Games.Cricket.# - Receives messages published to

‘Games.Cricket.SriLanka’ and ‘Games.Cricket.UK’


Distributed Queues

● Provides strict or best effort support for inorder delivery

● There are no guarantee about the global order seen across subscribers


Distributed Queues with MB

● Provides strict or best effort support for inorder delivery

● There are no guarantee about the global order seen across subscribers

● Supported Protocols AMQP/JMS, MQTT

MB Roadmap

• Support for AMQP 1.0 and JMS 2.0

• Improvements to Storage Scheme, Performance improvements

• Analytics for MB

• C5 migration

• Improved GUI

• MQTT + Websockets

Introducing WSO2 Data Services Server

Painpoints: ● Application Silos / Different data

stores● Cumbersome data access/

transformation logic● Business logic vs Data access logic● Repetitive Code

DSS:● Well defined interfaces (standard based)● Encapsulated data logic● Configuration driven● Loose Coupling between App & Data stores● Scale as a separate architectural layer.

WSO2 Data Services Server : Features

• Multiple Data Sources

• Nested Queries

• Batch Processing

• Distributed Transactions

• OData (www.odata.org)– Quickly expose data as REST APIs

– Standardized [URL conventions, Request/Response Headers,

Status Codes ,Payload formats]

http://www.odata.org

Demonstration - Use Case

• As part of an effort to improve traffic monitoring process, City transport authorities are making buses to send useful information while it's in operation such as speed, location data and direction.

• To ensure safety of passengers, officials would like to detect bus drivers who violate speed limits. Supervisors will investigate these incidents case-by-case basis.

• If a supervisor determines a particular incident is undue, bus driver will be placed under further investigation/review.

Demonstration - Solution Architecture

Demonstration - Concepts (OData)

Retrieve duty roster: GET http://localhost:9765/odata/AllocationData/DriverAllocations/duty_roster(1){ "@odata.context": "$metadata#duty_roster/$entity", "@odata.etag": "8d32bf38-8d8f-30cb-8e87-36ebeaedc2a9", "vehicleId": 1, "driverId": 1}

Retrieve employee information:GET http://localhost:9765/odata/HRDataAPI/records/employee(1){ "@odata.context": "$metadata#employee/$entity", "@odata.etag": "58bff0d2-09a4-3315-b2a0-667626dbf5c7", "id": 1, "supervisorId": 2, "username": "driver", "status": "UnderReview", "name": "Driver"}

Update Employee status:PATCH http://localhost:9765/odata/HRDataAPI/records/employee(1){ "status" :"UnderReview"}

http://localhost:9765/odata/AllocationData/DriverAllocations/duty_roster(1)

http://localhost:9765/odata/HRDataAPI/records/employee(1)

http://localhost:9765/odata/HRDataAPI/records/employee(1)

Demonstration - Concepts (Inbound Endpoints)<inboundEndpoint name="speed-alert-receiver" onError="fault" protocol="jms" sequence="BusSpeedViolationsSeq" suspend="false" xmlns="http://ws.apache.org/ns/synapse"> <parameters> <parameter name="interval">1</parameter> <parameter name="sequential">true</parameter> <parameter name="coordination">true</parameter> <parameter name="transport.jms.Destination">SpeedViolations</parameter> <parameter name="transport.jms.CacheLevel">3</parameter> <parameter name="transport.jms.ConnectionFactoryJNDIName">QueueConnectionFactory</parameter> <parameter name="java.naming.factory.initial">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</parameter> <parameter name="java.naming.provider.url">repository/conf/jndi.properties</parameter> <parameter name="transport.jms.SessionAcknowledgement">SESSION_TRANSACTED</parameter> <parameter name="transport.jms.SessionTransacted">true</parameter> <parameter name="transport.jms.SubscriptionDurable">false</parameter> <parameter name="transport.jms.ConnectionFactoryType">queue</parameter> <parameter name="transport.jms.ReceiveTimeout">100</parameter> <parameter name="transport.jms.ContentType">application/json</parameter> </parameters></inboundEndpoint>

<inboundEndpoint name="hr-request-receiver" onError="fault" protocol="http" sequence="HRSeq" suspend="false" xmlns="http://ws.apache.org/ns/synapse"> <parameters> <parameter name="inbound.http.port">8081</parameter> </parameters></inboundEndpoint>

Demonstration - Concepts (Endpoints)

<endpoint name="DutyRosterEP" xmlns="http://ws.apache.org/ns/synapse"> <http method="get" trace="disable" uri-template="http://localhost:9765/odata/AllocationData/DriverAllocations/duty_roster({uri.var.busId})"/></endpoint>

<endpoint name="GetEmployeeRecordsEP" xmlns="http://ws.apache.org/ns/synapse"> <http method="get" trace="disable" uri-template="http://localhost:9765/odata/HRDataAPI/records/employee({uri.var.employeeId})"/></endpoint>

<endpoint name="UpdateEmployStatusEP" xmlns="http://ws.apache.org/ns/synapse"> <http method="patch" trace="disable" uri-template="http://localhost:9765/odata/HRDataAPI/records/employee({uri.var.employeeId})"/></endpoint>

Usage:

<property expression="json-eval($.driverId)" name="uri.var.employeeId" scope="default" type="STRING"/><call> <endpoint key="GetEmployeeRecordsEP"/></call>

Demonstration - Concepts (Data Mapping)

<datamapper config="gov:datamapper/MapSpeedViolationResultToHRSystem.dmc"

description="Map-Speed-Violation-OutCome-To-HR-System"inputSchema="gov:datamapper/MapSpeedViolationResult

ToHRSystem_inputSchema.json" inputType="JSON"

outputSchema="gov:datamapper/MapSpeedViolationResultToHRSystem_outputSchema.json" outputType="JSON" />

Demonstration - Concepts (Payload Messages and Service Chains)

<sequence name="BusSpeedViolationsSeq" onError="fault" statistics="enable" trace="disable"

xmlns="http://ws.apache.org/ns/synapse">

<property expression="json-eval($.event.payloadData.id)"

name="uri.var.busId" scope="default" type="STRING"/>

<property expression="json-eval($.event.payloadData.id)"

name="busId" scope="default" type="STRING"/>

<property expression="json-eval($.event.payloadData.timeStamp)"

name="timeStamp" scope="default" type="STRING"/>

<property name="uri.var.busId" scope="default"

type="STRING" value="1"/>

<call>

<endpoint key="DutyRosterEP"/>

</call>

<property expression="json-eval($.driverId)"

name="uri.var.employeeId" scope="default" type="STRING"/>


name="driverId" scope="default" type="STRING"/>

<call>

<endpoint key="GetEmployeeRecordsEP"/>

</call>

Demonstration - Concepts (Payload Messages and Service Chains)

<property expression="json-eval($.username)"

name="driverUsername" scope="default" type="STRING"/>

<property expression="json-eval($.name)"

name="driverName" scope="default" type="STRING"/>

<property expression="json-eval($.supervisorId)"

name="supervisorId" scope="default" type="STRING"/>

<property expression="json-eval($.supervisorId)"

name="uri.var.employeeId" scope="default" type="STRING"/>

<call>

<endpoint key="GetEmployeeRecordsEP"/>

</call>

<property expression="json-eval($.username)"

name="supervisorUserName" scope="default" type="STRING"/>

Demonstration - Concepts (Payload Messages and Service Chains) <payloadFactory media-type="json">

<format> {"processDefinitionKey": "trafficViolations","businessKey": "trafficKey","tenantId": "-1234",

"variables": [

{"name": "vehicleId", "value": "$1"},

{ "name": "timeStamp", "value": $2 },

{ "name": "driverId", "value": "$3"},

{ "name": "driverUserName", "value": "$4"},

{ "name": "driverName", "value": "$5"},

{ "name": "supervisorUserName", "value": "$6" }]}

</format>

<args>

<arg evaluator="xml" expression="$ctx:busId" xmlns:ns="http://org.apache.synapse/xsd"/>

<arg evaluator="xml" expression="$ctx:timeStamp" xmlns:ns="http://org.apache.synapse/xsd"/>

<arg evaluator="xml" expression="$ctx:driverId" xmlns:ns="http://org.apache.synapse/xsd"/>

<arg evaluator="xml" expression="$ctx:driverUsername" xmlns:ns="http://org.apache.synapse/xsd"/>

<arg evaluator="xml" expression="$ctx:driverName" xmlns:ns="http://org.apache.synapse/xsd"/>

<arg evaluator="xml" expression="$ctx:supervisorUserName" xmlns:ns="http://org.apache.synapse/xsd"/>

</args>

</payloadFactory>

<property name="messageType" scope="axis2" type="STRING" value="application/json"/>

<property expression="fn:concat('Basic ', base64Encode('admin:admin'))" name="Authorization" scope="transport"

type="STRING" xmlns:ns="http://org.apache.synapse/xsd"/>

<call> <endpoint key="CreateBPMNWorkFlowEP"/> </call>

<drop/>

</sequence>

Demonstration - Concepts (Payload Messages and Service Chains)<sequence name="HRSeq" statistics="enable" trace="disable"

xmlns="http://ws.apache.org/ns/synapse">


name="uri.var.employeeId" scope="default" type="STRING" />

<datamapper config="gov:datamapper/MapSpeedViolationResultToHRSystem.dmc"

description="Map-Speed-Violation-OutCome-To-HR-System"

inputSchema="gov:datamapper/MapSpeedViolationResultToHRSystem_inputSchema.json"

inputType="JSON"

outputSchema="gov:datamapper/MapSpeedViolationResultToHRSystem_outputSchema.json"

outputType="JSON" />

<property name="FORCE_SC_ACCEPTED" scope="axis2" type="STRING" value="true" />

<property name="OUT_ONLY" scope="default" type="STRING" value="true" />

<call description="">

<endpoint key="UpdateEmployStatusEP" />

</call>

</sequence>

Demonstration - Concepts (BPMN Workflows)

Next generation Integration Platform

• WSO2 Gateway Framework– An ultra high performance, lightweight and reusable message

Gateway framework that encapsulates the messaging between source and target systems

– HTTP/s message gateway leveraging Netty, LMAX Disruptor and WSO2 Pass-Thru messaging architecture.

WSO2 Integration product suite

• Series of products based on WSO2 GW Core framework.

WSO2 Process Center

Process owner / analyst

Process participant

Developer

Create / improve / standardize Search / browse

/ follow

View

Develop executable processes

Executable process is associated with Process Center

Thank You!

#WSO2ConEU

Share your feedback for this session

wso2con.com/app

wso2 integration platform deep dive eu con 2016

Technology