Download - INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer
![Page 1: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/1.jpg)
INNOV-10Progress® Event Engine™Technical Overview
Prashant ThummaPrincipal Software Engineer
![Page 2: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/2.jpg)
2 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
![Page 3: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/3.jpg)
3 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Event Stream Processing (CEP)
ESP is about Operational Business Data– Capture– Query– Analysis
Characteristics– High Volume Streaming Data– Time Critical Analysis– Consistent and Reproducible Results– Highly Available
Simple Structured Data– Data is not Relational or complex Object model
What is ESP?
![Page 4: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/4.jpg)
4 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Event Stream Processing (ESP)
Event Processing Language– Data Driven– Flexible– Complete
Event Persistence Engine– High Performance
Analytical Capabilities– Temporal Data Analysis– Inbound and Outbound Event Processors
High Availability Modeling, Deployment and Development Tools
ESP Platform Requirements
![Page 5: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/5.jpg)
5 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Event Stream Processing (CEP)
Financial Markets– Program Trading– Testing Trading Strategies
RFID– Inventory Logistics– Asset Tracking– Theft Detection
Manufacturing– Capture and analysis of data from PLCs
Telecommunications– Cell Phone Call Tracking– Multimedia Services Billing– Network Management
Scientific– Sensor Data
ESP Applications
![Page 6: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/6.jpg)
6 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
![Page 7: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/7.jpg)
7 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress ESP Products
Event Cache for JMS
Adapter
Event Server
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
XML Configuration FilesJava API
Progress® Event Engine™ (PSEE)
C++ API
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
XML Configuration Files
Apama
![Page 8: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/8.jpg)
8 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
![Page 9: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/9.jpg)
9 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
It’s all about Event Data– Collection, storage, management and query
What is Event Data?– Streaming, constantly flowing information
Stock Trades, RFID, IP Packets, Mfg PLC’s– Categorized
Stock tick symbol, RFID tag ID, IP address– Typically Time Ordered
Trade time, RFID scan time
What is it for?
![Page 10: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/10.jpg)
10 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
3. Query
query
client client client
Event Cache
Service
1. CollectEvent Data
Feed store
Convert external events to PSEE events
2. OrganizeStructures data for fast retrieval
sort/mergePersistent Storage
disk disk
disk disk
Data Flow
![Page 11: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/11.jpg)
11 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
High Performance, Scalable– Collection Engine
Store tens of thousands events per second 20 TB+ database size Index data is ~1% of database size
– Query Engine High Performance Pipelined Architecture Extensible Functionality
– Simultaneous Query and Collection Writes do not block reads!
Why PSEE?
![Page 12: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/12.jpg)
12 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Transparent Data Management– Event Persistence
Requires no coding to persist events
– Transactional Consistent Data Hidden and Managed by PSEE
– Tunable Latency “Real-time” defined by you!
Why PSEE?
![Page 13: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/13.jpg)
13 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Simple API Supports Multiple Languages
– Java and C++ language support Multiple Platform Support
– Window, Solaris, Linux
– Heterogeneous deployment
Why PSEE?
![Page 14: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/14.jpg)
14 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Functor: A unit of work in PSEE– 3 Main types (Source, Process, Sink)– Built in library of Functors– Customer coded
Pipeline: A sequence of Functors– Collect, process, transform event streams
Task Manager– Manages pipeline execution
Events: Input and Output of Pipelines
Components
![Page 15: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/15.jpg)
15 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
PSEE Event– XML schema
– Supports All C++ Primitive Types
Event Definition
<EventDefinitions bundleName=“TABundle” …> <Event name=“Trade” type=“TimedCategorizedEvent”> <Member name=“price” type=“Float32”/> <Member name=“volume” type=“UInt32”/> </Event></EventDefinitions>
![Page 16: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/16.jpg)
16 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
class MyCollectorFunctor : public Functor {public:
void initialize(FunctorContext& ctx);void handle(FunctorContext& ctx);void finalize(FunctorContext& ctx);
private:// functor data variables
}
Progress Event Engine
Implements Event Stream Processing Logic Interface For Accessing Events
Functor
![Page 17: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/17.jpg)
17 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event EngineCollector Pipeline
Event-Data File
for Bulk Loading
Event
Stream
StoreRealtime
Functor
Real-Time
Data Feed
Series
(PSEE Database)
MyCollector
Functor
![Page 18: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/18.jpg)
18 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event EngineQuery Pipeline
Series
(PSEE Database)
Retrieve
Functor
Filter
Functor
Sort
Functor
Event
Stream
Event
Stream
Event
Stream
MyXMLwriter
Functor
![Page 19: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/19.jpg)
19 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Collector Functors– StoreRealtime
– StoreBulkload
Query Functors– Retrieve
– Filter
– Sort
– Count
– Group Count
Predefined Functors
![Page 20: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/20.jpg)
20 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event EngineScalable Deployment
OSES
Qu
ery
Ad
apte
r
PS
EE
OSES
Co
llect
ion
Ad
apte
r 2
Co
llect
ion
Ad
apte
r 1 PS
EE
Raw Series
Optimized db’s for query
Series database
OSES
Qu
ery
Ad
apte
r
PS
EE
Qu
ery
Ad
apte
r
OSES
PS
EE
VMMA & CFA Scalability
MessagingScalability
![Page 21: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/21.jpg)
21 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
![Page 22: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/22.jpg)
22 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Performance
Event storage (events/sec)– Real-time collector – pipeline using
StoreRealtime functor– Bulk load collector – pipeline using
StoreBulkload functor Event retrieval (events/sec)
– Threaded execution of query pipeline Random query Query on several (randomly chosen)
categories
Performance Metrics
![Page 23: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/23.jpg)
23 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Performance
Database Characteristics– 160 Million Events
– 20 Byte Event
– 10000 Categories
System Configuration– Sun Fire-V440
– Solaris 8
– 4 CPUs (1GHz)
– 8 GB Memory
Performance Testing Environment
![Page 24: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/24.jpg)
24 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Performance
Real-time collector– 94,062 events / second
Bulk load collector– 191,152 events / second
Query– 213 Queries / second
– 10000 events / query
– 2,134,560 events / sec
Performance Testing Results
![Page 25: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/25.jpg)
25 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
![Page 26: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/26.jpg)
26 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
D I S C L A I M E R
Under Development
This talk includes information about potential future products and/or product enhancements.
What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here.
D I S C L A I M E R
![Page 27: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/27.jpg)
27 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Integration
Provides Event Persistence for Apama Initial Implementation for Customer
– Tick Database for Back-Testing Trading Strategies
– Query Interface Using Apama Event Language
Future– Extend Apama Event Language to support
Persistence– Seamless Configuration via Apama
Management Interfaces
Integration with Apama
![Page 28: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/28.jpg)
28 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Integration
PSEE on the bus using Event Cache Event Cache Features
– Native JMS Message Based Query
– Results are Java Objects
– Batched for Optimal Performance
Integration with Sonic
![Page 29: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/29.jpg)
29 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Integration
PSEE based RFID Accelerator provides 4GL Interface
Web Services based Implementation RFID Accelerator as a reference
implementation Further 4GL Integration
Integration with 4GL
![Page 30: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/30.jpg)
30 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
In Summary
Exponential growth of event data requires ESP solutions
PSEE + Apama is an ESP solution
Based on proven technologies Simple! Infrastructure problems
are handled by PSEE It can be integrated into existing
business systems
![Page 31: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/31.jpg)
31 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Questions?
![Page 32: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/32.jpg)
32 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Thank you for your time!
![Page 33: INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer](https://reader031.vdocuments.us/reader031/viewer/2022020417/56649f355503460f94c53496/html5/thumbnails/33.jpg)
33 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation