imc summit 2016 innovation - steve wilkes - tap into your enterprise – why database change and imc...
TRANSCRIPT
Tap Into Your EnterpriseWhy Database Change and IMC Are an Ideal Match
Steve Wilkes – Striim Founder / CTO
See all the presentations from the In-Memory Computing Summit at http://imcsummit.org
EnterpriseApplicatio
ns
You All Have Enterprise Applications• Finance• eCommerce• Supply Chain• HR• Inventory• CRM• Manufacturing• Support
EnterpriseApplicatio
ns
And Those Applications Use Data-bases
• Oracle• MySQL• SQL Server• DB2• Postgres• HP NonStop• Mainframe
How Do You Tap Into That Data For IMC?
IMC
And Deliver Processed Data To Any-where?
IMC
Databases / ODS / EDWFiles For Up-Stream ProcessingMessage Queues / Kafka for Data As a ServiceCloud for Elastic Storage and ScalabilityHadoop / NOSQL for Data Lake
SQL?
IMC
SQLQueries
SQL Queries Against Production DatabasesAre Forbidden by Most DBAs
Replicas and SQL?
IMC
SQLQueries
Read Only Replicas are Expensive andRequire a lot of Extra Resources
Replicate
Use Change Data Capture (CDC)
IMC
CDC Enables Non-Intrusive Collectionof Streaming Database Change
CDC
Delete
Insert
When DML Operations Occur They Are Written to the Log and Captured as Events by CDC
Databases Write All Operations to a Set ofTransaction Logs for Recovery / Other Purposes
Instead of Using Timestamps or Triggers CDC Directly Reads These Logs and Collects Operations
Update
How Does CDC Work?
IMCCDC
DMLInser
tUpdateDelete
TransactionLogs
110100001101010011
Insert
Use CDC Only Without IMC?
CDC
Works for Certain Use-Cases But CDC DataOften Needs Pre-Processing Before It Is Useful
IMC
What IMC Processing Can You Do?
CDC
Filter Out Unnecessary DataAggregate to Remove Redundancyand Obtain Trends Over Time
Transform to the Format You Need
What IMC Processing Can You Do?
CDC
Enrich Streaming Data With Context LoadedInto IMDG Without Slowing Processing
What IMC Analytics Could You Do?
CDC
Once You Have Streaming CDC Data You CanPerform Complex In-Memory Analytics in Real-Time
Search for Patterns
Identify Anomalies
Correlate in Time& Space
Visualize & Analyze
Alert on Issues
Trigger BusinessProcesses / Workflows
Analytics
One Thing to Remember
Change Data Capture & Streaming Integration
should be part ofyour IMC & Enterprise
Data Strategy
