next-generation bpm - how to create intelligent business processes thanks to big data and apache...

Next-Generation BPM –

How to create intelligent Business Processes

thanks to Big Data Talend, Global Leader in Open Source Integration Solutions

Kai Wähner [email protected]

@KaiWaehner

Xing / LinkedIn

www.kai-waehner.de

© Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Consulting

Developing

Coaching

Speaking

Writing

Main Tasks

Requirements Engineering

Enterprise Architecture Management

Business Process Management

Architecture and Development of Applications

Service-oriented Architecture

Integration of Legacy Applications

Cloud Computing

Big Data

Contact

Email: [email protected]

Blog: www.kai-waehner.de/blog

Twitter: @KaiWaehner

Social Networks: Xing, LinkedIn

Kai Wähner


Key messages

BPM should be used (just) for optimizing business processes!

Intelligent business processes need big data and integration!

Big data will reduce human interactions in BPM further!


• Big data paradigm shift

• Use cases for big data

• Intelligent business processes

• Technology and product perspective

• Implementation of an use case

Agenda


William Edwards Deming (1900 –1993)

American statistician, professor, author, lecturer and consultant

“If you can't measure it, you can't manage it.”

Why should you care about big data?


„Silence the HiPPOs“ (highest-paid person‘s opinion)

Being able to interpret unimaginable large data stream, the gut feeling is no longer justified!

Why should you care about big data?


Why does big data exist?

Changing Interactions

Changing Scale

Changing Expectations

Cloud

Sensors


Three shifts in the way we analyze information

• Messiness: Using ALL data, not just samples • Also bad data (e.g. Word spell checker, Google auto-

complete and „did you mean...“ recommendation

• Correlations: Instead of causalities • May not tell us WHY something is happening, but THAT it

is happening • In many situations, this is good enough • What drug substance cures cancer? When should I buy an

airplane ticket?

• Datafication: Store, process, combine, reuse, enhance all data! • Digitalisation (Amazon Kindle Read) vs. Datafication

(Google Books Read, Search, Process, ...) • Words becomes data: Google books: not just read, but

also search, analyse, etc. • Locations becomes data: GPS: not just navigation, but

also insurance costs, economic routes, etc.


The Vs of big data

Volume

(terabytes,

petabytes)

Variety (social networks, blog posts, logs,

sensors, etc.)

Velocity

(realtime or near-realtime)

Value


Big Data Integration – Land data in a Big Data cluster

– Implement or generate parallel processes

Big Data Manipulation – Simplify manipulation, such as sort and filter

– Computational expensive functions

Big Data Quality & Governance – Identify linkages and duplicates, validate big data

– Match component, execute basic quality features

Big Data Project Management – Place frameworks around big data projects

– Common Repository, scheduling, monitoring

Big data tasks to solve - before analysis







Agenda


➜ A lot of data must be stored „forever“

➜ Numbers increase exponentially

➜ Goal: As cheap as possible

➜ Problem: (Fast) queries must still be possible

➜ Solution: Commodity servers and „Hadoop querying“

Global Parcel Service

http://archive.org/stream/BigDataImPraxiseinsatz-SzenarienBeispieleEffekte/Big_Data_BITKOM-Leitfaden_Sept.2012#page/n0/mode/2up

Storage Reduce costs


“The advantage of their new system is that they can now look at their data [from their log processing system] in anyway they want:

➜ Nightly MapReduce jobs collect statistics about their mail system such as spam counts by domain, bytes transferred and number of logins.

➜ When they wanted to find out which part of the world their customers logged in from, a quick [ad hoc] MapReduce job was created and they had the answer within a few hours. Not really possible in your typical ETL system.”

http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data

Replace ETL Improve performance


http://hkotadia.com/archives/5021

Deduce Customer Defections

Risk management Customer success


➜ With revenue of almost USD 30 billion and a network of 800 locations, Macy's is considered the largest store operator in the USA

➜ Daily price check analysis of its 10,000 articles in less than two hours

➜ Whenever a neighboring competitor anywhere between New York and Los Angeles goes for aggressive price reductions, Macy's follows its example

➜ If there is no market competitor, the prices remain unchanged

http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702

Flexible pricing Increase revenue


➜ How do you put this big data easily in the hands of the people that need it?

➜ Making the data “actionable” is the real challenge.

➜ Seeing the information that helps make a decision on a composite dashboard is just the first step and where too many companies stop.

➜ A business must be able to fire off the business process to execute the decision made regarding the data.

Great big data use cases, but ...

Kai Wähner

http://smartdatacollective.com/matt-davies/104576/data-driven-bpm-making-big-data-actionable







Agenda


➜ Humans have to interpret large data to make decision. Using gut feeling is nothing but gambling.

➜ Just doing big data analytics is not enough. Systematic and monitored human interactions are as important to get best outcomes.

➜ An intelligent business process

combines big data and BPM. This

enables humans to make data-driven

decisions based on big data analytics.

Intelligent business processes


➜ Process starts action (PULL Big Data)

• Manual or automated

• Faster responses (e.g. „spam by domain“)

• Better outcomes (e.g. „recommendation engine“)

➜ Data starts action (Big Data PUSH)

• (Usually) automated

• Predictive processes (e.g. „preventing flu epidemic“)

• Handle before it happens (e.g. „customer deduction“)

Intelligent business processes


➜ How are they related?

➜ How to combine?

➜ How to realize this technically?

Combination of big data and BPM


How BPM?

Kai Wähner

Script Task Service Task

Groovy

JavaScript

etc.

SOAP Web Service

Everything

from Cobol

to Ruby...

... or a

„big data

service“


Challenge

Kai Wähner

Separation of Concerns


Building Blocks for „Intelligent Business Processes“

Kai Wähner

Integration • ETL

• Connectivity / adaptors to connect to external

systems using a variety of different protocols

• Predefined EIP for message routing

Big Data • Processing

• Analytics

BPM • Do queries to make decisions

• Human or machine







Agenda


Kai Wähner

Integration • Extract Transform Load (ETL)





• Analytics





Enterprise Integration Patterns


AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials( S3Sample.class.getResourceAsStream("AwsCredentials.properties"))); String bucketName = "my-first-s3-bucket-" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceException ase) { // error handling... } catch (AmazonClientException ace) { // error handling... }

Do not write all that “glue code”!


// Producer

from(“ftp:toS3")

.setHeader(S3Constants.KEY, simple(“order.txt"))

.to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s)

// Consumer

from(„salesforce://orders__c?user=dummy1“)

.filter(„customer == ${dummyCustomer})

.to(“ibm-database:orderData")

Integration framework (e.g. Apache Camel)


Enterprise Service Bus (e.g. Talend ESB)


Connectivity

Routing

Transformation

Complexity

of Integration

Enterprise Service Bus

Integration Suite

Low High

Integration

Framework

INTEGRATION

Tooling

Monitoring

Support

+

BUSINESS PROCESS MGT.

BIG DATA / MDM

REGISTRY / REPOSITORY

RULES ENGINE

„YOU NAME IT“

Alternatives for integration

+


Kai Wähner

Integration • ETL





• Analytics





Technology perspective

How to process big data?


The defacto standard for big data processing



Even Microsoft (the .NET house) relies on Hadoop since 2011


“A big part of [the company’s strategy] includes wiring SQL Server 2012 (formerly known by the codename “Denali”) to the Hadoop distributed computing platform, and bringing Hadoop to Windows Server and Azure”


Apache Hadoop, an open-source software library, is a framework that allows for the distributed processing of large data sets across clusters of commodity hardware using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

What is Hadoop?


MapReduce

HDFS

Ecosystem

Features

included

Hadoop Distribution

Integration Suite

few many

Apache

Hadoop

Packaging

Deployment-Tooling

Support

+ Tooling / Modeling

Code Generation

Scheduling

Other Tools (ESB, BPM, ...)

+

Hadoop alternatives


Hadoop cannot solve every big data problem.

Complex event processing and real-time analytics have to be solved in another way (at least today).

In-memory computing and streaming platforms are good alternatives or complements to Hadoop for processing

and analyzing big data.

(Near) Realtime?

!


Kai Wähner

Integration • ETL





• Analytics





Standards

Kai Wähner

BPM

BPEL BPMN

XPDL

WF-XML

BPEL4People

jPDL

ARIS EPC


BPMN

Kai Wähner

„Business Process Model and Notation (BPMN) is a graphical

representation for specifying business processes in a business process

model.“ BPMN 2.0 is also executable!

Wikipedia


Alternatives for BPM

Kai Wähner

Complexity of

Orchestration Low High

BPM Framework

BPM Suite

Integration Suite

Coding

Service Tasks

Human Interaction

GUI

BPM

Tooling

Monitoring

Support

+

ESB

BIG DATA / MDM

REGISTRY / REPOSITORY

RULES ENGINE

„YOU NAME IT“

+


Kai Wähner

Integration • ETL





• Analytics



Let‘s realize

it !!!



Frameworks vs. Tools

Kai Wähner

Complexity of


Frameworks

Specific Tools

Suite of Tools

e.g.

Talend Unified Platform

i.e ALL-IN-ONE

(Integration, Big Data, BPM)

e.g.

Camel (Integration)

Hadoop (Big Data)

Activiti (BPM)

e.g.

Mule ESB (Integration)

MapR (Big Data)

Camunda (BPM)


Custom combination of integration, big data and BPM?

Kai Wähner

• A lot of glue code

• Testing

• Bugfixing

• No support

Some other people already had

the problems you would have!







Agenda


➜ With revenue of almost USD 30 billion and a network of 800 locations, Macy's is considered the largest store operator in the USA

➜ Daily price check analysis of its 10,000 articles in less than two hours

➜ Whenever a neighboring competitor anywhere between New York and Los Angeles goes for aggressive price reductions, Macy's follows its example

➜ If there is no market competitor, the prices remain unchanged

http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702

Flexible pricing Increase revenue


Implementation of an use case

Complexity of


Frameworks

Specific Tools

Suite of Tools

e.g.


i.e ALL-IN-ONE

(Integration, Big Data, BPM)

e.g.

Camel (Integration)

Hadoop (Big Data)

Activiti (BPM)

e.g.

Mule ESB (Integration)

MapR (Big Data)

Camunda (BPM)


Data

Quality Data

Integration MDM ESB

Big

Data

Big Data

Open source license

Free of charge

Optional support

Based on open source projects such as Eclipse or Apache Camel, CXF, Hadoop

Commercial license

Subscription model

Support included

Data

Quality

Data

Integration MDM ESB BPM



Example (Talend): Integration

Connect to data sources from competitors, for example via REST service, Twitter API, or custom scripts.


Example (Talend): Big Data Processing

Move data to HDFS for processing, as your classic servers and data warehouses are not able to process this semi-structured data fast enough (and cheap), probably. Manipulate the data, in other words, filter relevant information, sort it, and compare it to prices of your products.


Example (Talend): Business Process

Start a new instance of a business process to review the result and continue with further tasks, such as calling a web service which does the price reduction in selected locations. Reviews can be done by human interaction or via automated tasks depending on the proposed price reduction.


„Talend Unified Platform“ in action...

Implementation of an use case


Did you get the key message?


Key messages

BPM should be used (just) for optimizing business processes!

Intelligent business processes need big data and integration!

Big data will reduce human interactions in BPM further!


Did you get the key message?

KAI WÄHNER [email protected]

www.kai-waehner.de

LinkedIn / Xing

@KaiWaehner

Thank you for your attention. Questions?

next-generation bpm - how to create intelligent business processes thanks to big data and apache...

Technology

big data talend

big data tasks

big data cluster

bad data

lot of data

big data match component

unimaginable large data

vs of big datavolume