lucas jellema oracle openworld 2014, san francisco, ca, usa how fast data is turned into fast...

Lucas Jellema

Oracle OpenWorld 2014, San Francisco, CA, USA

How Fast Data Is Turned into Fast Information and Timely Action

2

Audience Challenge

3

Audience Challenge

4

Audience Challenge

5

Audience Challenge

6

Audience Challenge

7

Audience Challenge

8

Filter

Pattern Detection

Agregate

9

Fast Data Example

14,016,114,116,116,013,114,016,013,113,014,116,014,113,014,116,013,114,0

Smart Processing• Information• Conclusion• Alert• Recommendation• Action

10

Demonstration: Live Tennis

• Tennis Tournament• Many matches played in parallel• The data that is produced:

– At a rate of up to 10 events/minute

Match Id, Player [who scored]14,016,114,116,116,013,114,0

11

Demonstration: Live Tennis

• The information, conclusions &actions we are looking for:– Scoreboard per game, set, match– Match start and completion (action:

inform next players for that court)– Interrupted match (action: go and check

out the reason for the interruption)

Fast Data

Smart Processing

• Scoreboard• Match start and

completion • Interrupted match

12

Real Time – from event to UIPush through Web Sockets

Fast Data

Smart Processing

Oracle Event Processor

WebLogic

eventWebSocket

Servermsg

msg

CQL queries

13

WebSocket Powered Scoreboard

14

OEP application to process fast tennis data

• Preparation– Define event definitions– Create local, in memory cache with static, enriching data– Gather (in this case generate) tennis data through adapter– Create Event Sink to consume all findings and publish to console

TennisMatchEvent

matchIdplayer

15

Match Level events

16

Rally’s to games

- The first player to have won more than 4 points

- and have won two or more points more than his opponent

TennisMatchEvent

matchIdplayer

17

Detect interrupted matches by ‘finding’ missing events

• When a match is interrupted, obviously no more ‘rally point events’ are produced

• Detecting the absence of these events for a match [that has begun] is equivalent to detecting an interruption of the match– Unless the match is complete because someone won

18

Detect interrupted matches by ‘finding’ missing events

19

Complete EPN diagram for Tennis Tournament Processor

• A single OEP application that consumes fine grained rally point events and performs three-stage aggregation and enrichment

TennisMatchEvent

matchIdplayer

New Match

Match Finish

InterruptedMatch

Set WonGame

Won

Overview

• What is [special about] Fast Data?– Continuous, Volume|Velocity|Variety, Real Time

• Challenges– Volatile, non persistent– Data => Information, Conclusion, Alert, Recommendation, Action

• Strategies– Smart gathering– Discard – filter, aggregate, pattern

(and also look for missing events!)– Promote (process, enrich)– Visualize

• Technology/Tools• Demonstration/Cases

21

Fast Data

• Tweet• Feed• Beat• Signal• Measurement• Message• Mail• Notification• Tick• Pulse

22

New theme (that brings it all together)

23

Some event producing devices

24

Most of these events….

26

27

Fast Data Processing

Fast Data

Smart Processing

• Information• Conclusion• Alert• Recommendation• Action

28

Fast Data ProcessingMulti-stage cleansing & aggregation

Fast Data

Smart Processing

• Information• Conclusion• Alert• Recommendation• Action

29

Typical Flow and Additional Challenge…

Business event

Bu

sin

ess

Val

ue

Data captured

Analysis completedAction taken

Fragmented event entities

TIME

30

The V-factor

VolumeVelocityVariety

VALUE

31

Key strategy

• Discard – as early as possible (close to the source)– Ignore irrelevant events– Filter out unneeded attributes– Takes samples instead of entire stream– Aggregate: merge multiple, correlated events into one

32

Fast Data Processing:Oracle Event Processor

Fast Data

Smart Processing

Oracle Event Processor

RMIFile

RESTHTTP Channel

JMSDatabase

Custom (Java)SOA Suite EDN

CoherenceJMX

QuickFix (financial)

RMIFileRESTHTTP ChannelJMSBusiness RuleDatabaseCustom (Java)SOA Suite EDNCoherenceJMX

33

Oracle Event Processor

• Light weight, real-time (sub-sub-second), in-memory, continuous query engine– Available in embedded form – with corresponding licence

• Interacts with many different channels – inbound and outbound• Has internal caches to enrich events and temporarily retain events• Uses CQL to:

– Filter, aggregate, enrich and detect patterns (including missing events)

events

Event Processor

35

Fast Data ProcessingFusion Middleware Tooling

Fast Data

Smart Processing

Oracle Event Processor

Coherence

SOA Suite 12c EDN

RMIFile

RESTHTTP Channel

JMSDatabase

JMXCustom (Java)

RMIFileRESTHTTP ChannelJMSBusiness RuleJMXDatabaseCustom (Java)

36

Fast Data ProcessingFusion Middleware Tooling

Fast DataSmart Processing

• Information• Conclusion• Alert• Recommendation• Action

OEP

BAM

ADFCoherence SOA Suite

EDNBPMSuite

BPELTask

BIRTD

ODIGolden Gate

NoSQL

37

Business User Friendly Exploration of Fast Data: Stream Explorer

38

Credit Card Theft Detection

• Several situations in the past– Credit card is stolen in the main terminal building– Several purchases are made in shops on the way from that area to the main exit

• Purchases between $200-$500 dollar• Purchases made within 5 minutes of each other• Sometimes the purchases are made in not entirely the direct route to the exit

EXIT

Main Terminal

39

Credit Card Theft Detection

• Several situations in the past– Credit card is stolen in the main terminal building– Several purchases are made in shops on the way from that area to the main exit

• Purchases between $200-$500 dollar• Purchases made within 5 minutes of each other• Sometimes the purchases are made in not entirely the direct route to the exit

• To catch the perpetrator– Consume the credit card purchase event stream for airport shops– Spot situations where three or more purchases of $200-$500 are made within 5

minutes from each other and roughly in the terminal => exit physical order– Publish an event to alert security staff

• To watch for any further purchases with that credit card• To inform show staff for that credit card• To send staff to the exit to try and apprehend the thief

(perhaps based on the shopping bags he is carrying from the shops he bought stuff at)

40

Catch me if you can

EXIT

Main Terminal

41

Catch me if you can

EXIT

Main Terminal

$440$300

$380

$250

42

CQL to detect ‘funny string of transactions’

43

Real Time – from Event to TaskOEP => SOA Suite 12c EDN

Fast Data

Smart Processing

Oracle Event Processor

SOA Suite 12c

EDNevent event

44

Real Time – from Event to TaskOEP => SOA Suite 12c EDN

Fast Data

Smart Processing

Oracle Event Processor

SOA Suite 12c

EDN

BPEL Task

BPMN

Medi-ator

event event

event

45

From OEP finding to EDN Business Event triggering the SOA Suite

46

Human consumers

• Slow at data processing• Not electronically connected• Visually oriented (1 picture > 1000 words)• Frequently (though perhaps decreasingly so) the actor or decision maker

• Interact along human communication channels• Use visualization to present findings, conclusions, recommended actions

– And as a second tier of fast data processing:Highlight (filter), aggregate, patterns, extrapolate/interpolate, missing elements

• Sometimes take over from humans and just take action

47

Visualize and Aggregate

48

Real Time – from event to UIBusiness Activity Monitoring

Fast Data

Smart Processing

Oracle Event Processor

WebLogic

JMS

event

msgBAMmsg

49

Real Time – from event to UIADF DVT Visualizations

Fast Data

Smart Processing

Oracle Event Processor

WebLogic

JMS

event

msg

ADF DVT

msg

50

Visualize physical locations of [string of] suspicious transactions

51

Summary

• Fast Data (events): Vast, Continuous, Velocity, Variety– Wanted: Near real time conclusions, recommendations, alerts, actions

• Strategy:– Discard – as early as possible (Filter, Aggregate)– Enrich, Correlate and Pattern Match, Missing Events, Retain, Publish higher level,

more coarse grained business event– Repeat this cycle multiple times (such as rally point, game, set, match)

• Technology for Fast Data processing: Oracle Event Processor & CQL– Interacts with JMS, EDN, RMI, HTTP (/REST), JMX, Database, Coherence

• New: Stream Explorer – business friendly, industry pattern based fast data explorations and visualization

• To assist humans in Fast Data and Information Processing: Visualization– Filter, Aggregate, Enrich, Pattern Match (1 picture > 1000 words)– Technology: BAM (Dashboard and Rule processing), ADF Data Visualization– Also: turn findings into actions using Human Task, BPEL and BPM via the SOA Suite

12c Event Delivery Network (EDN)

53

Fast Data Example

14,016,114,116,116,013,114,016,013,113,014,116,014,113,014,116,013,114,0

Smart Processing

Oracle Event Processor

54

Demonstration: Live Tennis

• Tennis Tournament• Many matches played in parallel• The data that is produced:

– At a rate of up to 10 events/minute

Match Id, Player [who scored]14,016,114,116,116,013,114,0

55

Demonstration: Live Tennis

• The information, conclusions &actions we are looking for:– Scoreboard per game, set, match– Match start and completion (action:

inform next players for that court)– Interrupted match (action: go and check

out the reason for the interruption)

Fast Data

Smart Processing

• Scoreboard• Match start and

completion • Interrupted match

56

OEP application to process fast tennis data

• Preparation– Define event definitions– Create local, in memory cache with static, enriching data– Gather (in this case generate) tennis data through adapter– Create Event Sink to consume all findings and publish to console

TennisMatchEvent

matchIdplayer

57

Simple Time-slice Aggregation

• Produce aggegrates once every 30 seconds– Count number of matches going on currently (meaning: in the last 30 seconds)– Calculate average time per rally (over the last 30 seconds)– Count total number of points played (over the last 30 seconds)

60

Match Level events

61

Rally’s to games

- The first player to have won more than 4 points

- and have won two or more points more than his opponent

TennisMatchEvent

matchIdplayer

62

Games to Sets

- The first player to have won more than 5 games

- and have won two or more games more than his opponent

63

Detect interrupted matches by ‘finding’ missing events

• When a match is interrupted, obviously no more ‘rally point events’ are produced

• Detecting the absence of these events for a match [that has begun] is equivalent to detecting an interruption of the match– Unless the match is complete because someone won

64

Detect interrupted matches by ‘finding’ missing events

65

Complete EPN diagram for Tennis Tournament Processor

• A single OEP application that consumes fine grained rally point events and performs three-stage aggregation and enrichment

TennisMatchEvent

matchIdplayer

New Match

Match Finish

InterruptedMatch

Set WonGame

Won