Dealing with Common Data Requirements in your Enterprise

Nipun SuwandaratnaSenior Solutions Engineer - WSO2

WSO2 Solution Architecture Best Practices Webinar Series - 2016

Agenda● Organizational Data

● Common Data Challenges of Modern Organizations

● Integrating with Different Messaging Infrastructures

● Data Services

● Data Analytics & Visualization

● High Availability

● Q&A

Organizations & Data

Old School

DB

Sys

tem

s

Files

Organization

Modern Data Ecosystem

External Systems

External Users

Organizational Data

● Master dataEg: Customer data, employee records, Supplier details, Product related data etc.

● Transactional dataThe data that master data participates in… transactions, discounts on bills etc. (changes constantly)

● Meta-dataData about data

Image: thinkpublic/photopin cc

Common Data Challenges Organizations Face● Work with multiple Data Transports and Data Formats

● Data Transformation and Validation

● Exposing data as services

● Secure and managed data access

● Federated data stores

● Data/Entity Aggregation

● Data Analytics

● Visualization of Data

Data Transports & FormatsFormats of data, their storage and transport mechanisms vary among

different systems

● Transports: HTTP, HTTPS, FTPS, SFTP, TCP, UDP, WebSocket, POP, IMAP, SMTP, JMS, AMQP, MQTT

● Formats & protocols: JSON, XML, SOAP, WS-*, HTML, EDI, HL7,Text, JPEG, MP4, binary formats

Integration

Integrating with Messaging Infrastructures

Message Transformation● Protocol and Format conversion and Message Translation

○ eg: SOAP to REST and XML to JSON and translate the output from one system to match the input format required by the other system

● Enrich Content ○ eg: Add or remove data fields; may require accessing a separate data source

● Wrap Content○ eg: Include additional message header fields or encryption source to query

required data● Data Validation

○ eg: Validate input data against a schema

Enterprise Service Bus

Message Transformation ExampleProtocol / Content-Type Conversion

Data Services

Exposing Data-As-ServicesWhy ?

● Decouple data from the infrastructure and the data sources and expose them through standard web services interfaces.

● Ability to incorporate multiple data sources/entities into a single data model (Data Federation)

Secure & Managed Data Access

● Transport and Application level security

● Authentication, authorization, confidentiality, integrity and encryption - with HTTP(S)

Basic Auth, WS-Security, WS-Trust, WS-SecureConversation, WS-Policy,

WS-Policy Attachment and WS-SecurityPolicy

● Authorization deals with defining who can access what

● Role based access control

● Fine-grained authorization with XACML

● Throttling access to data

Federated Data Stores● Expose data from multiple data sources through a single service

● Facilitates entity aggregation

Data/Entity Aggregation

WSO2 Data Services Features

Ref: http://wso2.com/products/data-services-server/

Analytics

Data Analytics● Batch Analytics

Analyze a set of data collected over a period of time. Suitable for high volumes of data.

● Real-Time AnalyticsContinuous processing of input data in real time. Suitable for critical systems where immediate actions is required e.g: Flight radar systems

● Interactive AnalyticsObtaining fast results on indexed data by executing ad-hoc queries

● Predictive AnalyticsPredict future events by analyzing historical and current data

Big Data

What is Big data ?

“Big data is a term for data sets that are so large or complex that traditional

data processing applications are inadequate to deal with them” - Ref: Wikipedia

Big Data AnalysisWhy ?

● Make informed Business decisions - make decisions based on patterns emerging from analyzing historic data

● Improve customer experience - discover customer preferences, purchasing patterns and present the most relevant data

● Process Improvements - identify areas of the business process that needs improvement

Big Data Analysis ExampleBetter customer experience in airline seat reservation/allocation

img ref: http://staticcontent.transat.com/airtransat/infovoyageurs/content/EN/seating-plan-a310-300(1).png

Real Time Analytics

● Identify most meaningful events within an event cloud

● Analyze the impact

● Acts on them in real time

Real Time Analytics ExampleCity Transport Control System - Analyzing traffic, monitor movement of busses, generate alerts based on traffic, speed & route

Predictive Analytics & Machine LearningApproaches:

● Machine LearningMachine learning is the science of getting computers to act without being explicitly programmed - http://online.stanford.edu/

● Other approaches such as statistical modeling

Predictive Analytics Examplee-Commerce sites use predictive analytics to suggest the most relevant merchandize, increasing sales opportunity

WSO2 Data Analytics

Ref:wso2.com

Data Visualization

Data Visualization Contd.

What is Data Visualization ?

● View data in a constructive and comprehensible format

● Facilitates interaction with data - drill into the data for visual

analysis

● Detect patterns (e.g: sales patterns) that may go un-noticed unless

data is properly visualized

High Availability

High Availability of Data

CONTACT US !