oracle coherence - events and real-time bi nrw... · failover and load-balancing functionality ......

Basel · Baden · Bern · Lausanne · Zurich · Düsseldorf · Frankfurt/M. · Freiburg i. Br. · Hamburg · Munich · Stuttgart

Oracle Coherence - Events and Real-Time BI

Peter WelkenbachTrivadis GmbH2007

11.03.2008Oracle Coherence 2 © 2007

TOC

The Situation Real time business response

The Solution EDA event driven architecture ESP event stream processing CEP complex event processing XTP extrem transaction processing

The Product Coherence ESP with Esper and Coherence

We deliver theplatform forperfectinteraction.


The Situation – Business Point of View

Business process management andautomation process monitoring, BAM, reporting

exceptions

Finance algorithmic trading, fraud detection, risk

management

Network and application monitoring intrusion detection, SLA monitoring

Sensor network applications RFID reading, scheduling and control of

fabrication lines, air traffic

John Bates, Progress Software

In the algorithmic trading space, an"algorithm" describes a sequence ofsteps by which patterns in real-timemarket data can be recognized and

responded to..

Source: www.progress.apama


Observe the news

Some News RSS (Really Simple Syndication):Include external

events

Requires sophisticatedtext processing,

computing intensive

Dow Jones: elementized news


The Challenge – Technical Point of View

High throughput - applications that process large volumes ofmessages (between 1,000 to >100k messages per second)

Low latency - applications that react in real-time to conditionsthat occur (from a few milliseconds to a few seconds)

Complex computations - applications that detect patterns among events, filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.

Find a fitting model Define Events, Design Events, ensure performance….


Reference Architecture with Coherence

Continous Queries (in-memory materialzed views) firing Events

Extrem scalability and distributed parallel computing with extremsimple programming model

Analyse event streams, corelate over time with CEP-Engine


Extrem complex event stream processing

Tranform,Enrich

CoherenceNamedCache A

CEP Engine (correlate and extract complex events)

CoherenceNamedCache B

MapListener MapEventFilter

Adapter-Bean

has

Notify

Publish

Start BusinessProcess Invoke Service

Externalevent

streams

Eventcloud


Event-Driven Architecture

a software architecture supporting the Production (Feed, Sensor,..), Detection (complex rule based pattern matching), consumption of and reactionto events

e.g. fire if a news feed informsabout “MyCompany” but in USA

“MyCompany == TheOneAndOnlyCompany”


EDA complements SOA

SO

A

Service at the core:Expose and handle

Event at the core:when x then y


What is an event?

Events observe a change in state A stock tick The receipt of a credit check request A password change A sensor measuring temperature every 10 msec A service response time for the last request

Represented in the system XML (e.g. RSS) Plain Old Java™ Object Key-value pairs

An event is a piece of datathat represents that

something happened inthe real world, or in

software system

Events flow instreams


Oracle Coherence – the Product

The product Features What is it good for

Cache-topologies Reliability Performance

Grid functionality EntryProcessors WorkManager



Coherence

Coherence is a distributed data cache,

typically in main memory across multiple nodes, that ensures data is “closer” to the applications

ability to issue the same transaction locally to multiple nodes,with recombination of the result at the primary

ability to copy cache data from a “hot spot” to another nodedynamically


Coherence - Reliability

Coherence’s load-balancing/reliability technology is a flexibleprimary/multiple-backup scheme.

Each time a node goes down, instead of doing rollback/roll-forward or recovery from a backup, the system reconstitutes the data from all the backup nodes, creates a new primary, and continues running

The primary/multiple-backup scheme allows greater ability toload-balance by choosing to access a primary or backup copy.

Coherence supports cluster-durable (rather than disk-durable)storage of data and transactions


What is it Good For ?

Isolation of Database

Data Caching connects local data with local computing for analysis in real time,

Distributed Processing (Grid) automatic and dynamic partitioning of data in distributed memory guarantees permanent data availability and transaction integrety failover and load-balancing functionality in-memory-grid and parallel transaction-handling

Event Driven Processing in-memory-grid and parallel transaction-handling and event-handling


Simple application

CacheFactory.ensureCluster();try {

NamedCache myCache = CacheFactory.getCache("test2");Person peter = new Person();

peter.setPrename("peter");peter.setSurname("welkenbach");peter.setGender("male");

myCache.put( "peter", peter);

Person p = (Person) myCache.get("peter");

System.out.println( p );

} finally {

CacheFactory.shutdown();

}


Cache Topologies



Replicated Topology

What: Brute-force implementation

of Clusterd Caching

Why: Designed for extreme read

performance

How: Replicate and maintain copies of all Entries in all Members Zero latency access as all Entries are local to members Replication process transparent to developer

Configurable Expiration Policies: LFU, LRU, Hybrid (LFU + LRU), Time-based, Never, Pluggable

Source: coherence user guide


Partitioned Topology

What: Sophisticated approach for

coherent scalable clusteredcaching

Why: Designed for

extreme scalability

How: Transparently partition, distribute and backup cache entries across

members

Often referred to as ‘Distributed Topology’

Configurable Expiration Policies: LFU, LRU, Hybrid (LFU + LRU), Time-based, Never, Pluggable



Near Topology (L2, L1 cache)



Caching Strategies

Coherence supports transparent read-write caching of anydatasource, including databases, web services, packagedapplications and filesystems,

the cache and database are kept fully synchronized.

To accomplish this, Coherence supports Read-Through, Write-Through, Refresh-Ahead and Write-Behind caching.

CacheStore is an application-specific adapter used to connect acache to a underlying datasource The CacheStore implementation accesses the datasource via a

custom data access mechanism


Coherence Deep Dive



Be lock free

Adopt queuing approach to serialize work requests Locking forms queues... so just make queues! Use a ‘command pattern’ + workers Read about lock-free architectures (they work)

Lock free means avoid transactions!

Transactions are implemented with Locks!

Use finite state machines to model system states

Use retrospective / recoverable transactions

Tangosol introduced the concept of EntryProcessors and WorkManager (JSR-237) to address the locking issue


Be lock free

Source: tangosol presentation


Execution

Targeted Execution

Parallel Execution

Data-Grid-Wide Execution

query-based Execution

Object oResult = map.invoke(key, agent);

Map mapResults = map.invokeAll(collectionKeys, agent);

Map mapResults = map.invokeAll((Filter) null, agent);

Map mapResults = map.invokeAll( filter, agent );


AGGREGATION

Double DResult =(Double) cache.aggregate( (Filter) null,

new DoubleSum("getPrice"));

All aggregators thatcome with Coherence

are parallel-aware

Source: tangosol presentation


Transactions

TransactionMap cache = CacheFactory.get(“test”);

Try {

Cache.begin()

// ….. Do some logic

Cache.commit()

} catch(Exception e) {

cache.rollback();

}


Queryable

NamedCache mapTrades = ...

Filter filter = new AndFilter(new EqualsFilter("getTrader", traderid),new EqualsFilter("getStatus", Status.OPEN)

);

Set setOpenTrades = mapTrades.entrySet( filter );

Query for all open orders for a particular trader with traderid


Continuous Query

NamedCache mapTrades = ...

Filter filter = new AndFilter(new EqualsFilter("getTrader", traderid),new EqualsFilter("getStatus", Status.OPEN)

);

NamedCache mapOpenTrades =new ContinuousQueryCache( mapTrades, filter);


Filtering Events

Coherence does support constructing filters over POJOs in aClustered Distributed Cache or Data Grid,

maintaining continuous real-time views and

using said filters for real-time event delivery, including multi-platform support (Java, .NET with C++ arriving in 2007).

public class DeletedFilter implements Filter, Serializable {public boolean evaluate(Object o) {

MapEvent evt = (MapEvent) o;return evt.getId() == MapEvent.ENTRY_DELETED;

}}

cache.addMapListener(new EventPrinter(),new DeletedFilter(), false);


Advanced: Listening to Queries

NamedCache trades = ...Filter filter =

new AndFilter(new EqualsFilter("getTrader", traderid),new EqualsFilter("getStatus", Status.OPEN));

Set setOpenTrades = trades.entrySet(filter);

For example, in a trading system it is possible to queryfor all open "Order" objects for a particular trader

To receive notifications of new trades being opened for that trader,closed by that trader or reassigned to or from another trader,the application can use the same filter

// receive events for all trade IDs that this trader is interested inmapOpenTrades.addMapListener(listener,

new MapEventFilter(filter), true);


Triple Bottom - Triple Bottom Pattern

A triple bottom is a reversal pattern with bullish implicationscomposed of three failed attempts at making new lows in thesame area, followed by a price move up through resistance.

This pattern is rare, but a very reliable buy signal

Source: http://www.thestockbandit.com/Triple-bottom.htm


How do I detect the absence of an event?

The below pattern fires if an event A is not followed by an event Bwithin 10 seconds.

select * from pattern[every EventA -> (timer:interval(10 sec) and not EventB)]


Distributed Data Grid and CEP-Engine Architecture

Tranform,Enrich

Eventcloud

Basel · Baden · Bern · Lausanne · Zurich · Düsseldorf · Frankfurt/M. · Freiburg i. Br. · Hamburg · Munich · Stuttgart

Thank your for your attention

www.trivadis.com

oracle coherence - events and real-time bi nrw... · failover and load-balancing functionality ......

Documents