Patterns for Deploying Analytics in the Real World
Sriskandarajah Suhothayan (Suho)Technical LeadWSO2
What’s Analytics ?
Problems to think about
• Can it handle my load ?• How costly it is ?• Adaptability ?• Can it analyse 3rd party systems ?• etc ...
Where to start ?
Where to start ?
• Think Big !
Where to start ?
• Think Big !
But...
• Start simple !• Eat Your Own Dog Food • Analyse what you already have
Step 1 :
Find Data Inside Your Organisation...
Collect Data Internally
• Don’t worry about – Data formats – Data sources – Platforms – Protocols
Start with WSO2 DAS it has a unified data capturing framework !
Deployment for Data Collection
Step 2 :
Understand how things have been ...
Deployment for Data AnalyticsBatch & Interactive Analytics
• Enable Searchability – Full text data– Drill down search
• See what has happened – Summarise the Data – Understand patterns and behaviors
Deployment for Data AnalyticsBatch & Interactive Analytics
• Enable Searchability – Full text data– Drill down search
• See what has happened – Summarise the data – Understand patterns and behaviors
• Simple Deployment – 2 Nodes– Use RDBMS to store the data
Deployment for Data AnalyticsBatch & Interactive Analytics
2 Node Deployment
Step 3 :
Keep informed ...
Deployment for Data AnalyticsRealtime Analytics
• Keep informed – Dashboard – Alerts– Feedback loops
Deployment for Data AnalyticsRealtime Analytics
• Keep informed – Dashboard – Alerts– Feedback loops
• High Availability – Zero downtime– Zero data loss
Realtime High Availability Deployment
Minimum 2 nodesMax throughput == 1 Node throughput
Deployment for Data CommunicationAlerting & Communicating
Legacy & Internal Services
Realtime + Batch Analytics
• Filter Data before you store – Realtime → Store & Process
• Summarize and store – Realtime → Store & Process
• Cross check with history – Lambda Architecture– Graph with Batch & Realtime
• Alerts based on batch processing – Batch → Realtime
From Batch
From Realtime
Step 4 :
Think ahead ...
Deployment for Data AnalyticsPredictive Analytics
1 Node of WSO2 ML 1 Node of WSO2 ML
Minimum High Availability Deployment
All you need a 2 Node
Deployment
Step 5 :
Expanding as a Connected Business …
Deployment for Data Collection ...From 3rd Party Apps & Cloud
HTTP
Utilize API Analytics !
Analyse Business with API Analytics
• APIs involved • Who invokes the APIs• Extract business information from
– Payloads– Resources URIs
Monetize APIs !
Step 6 :
Scale with your Data ...
Scaling Analytics Deployment The Changes !
• Realtime – Supported by Apache Storm
• For High Memory Requirement or CPU Intensive Processing – No query changes
• Batch – Move from RDBMS to HBase/Cassandra
• WSO2 DAS have a Data Abstraction Layer • Independent of underlying Data Store
Seamless migration :)
Realtime Scalable Deployment ...
Event Processing offloaded to Siddhi Running on Apache Storm Seamlessly :)
Realtime Scalable Deployment ...
Handling Stateless & Stateful Queries
Realtime Scalable Deployment
Apache Storm Cluster + N CEP nodes
Deployment for Scalable Data Analytics
Minimum 8 Nodes Deployment
(+ Storm if needed)
Step 7 :
Sense the world around you ...
Deployment for Data CollectionFrom Sensors
Analytics on the Edgewith WSO2 Siddhi
Push
Deployment for Data CommunicationMobile & 3rd Party Apps
● Expose analytics results as API
○ Mobile Apps, Third Party
● Provides ○ Security, Billing, ○ Throttling, Quotas & SLA
● How ? ○ Write data to database from DAS ○ Build Services via WSO2 Data Services Server or use Analytics
REST API○ Expose them as APIs via WSO2 API Manager
Analytics Life Cycle
Predefined analytics
• Artifacts bundled as CApps to and moved
Dev → Test → Preprod → Prod
Analytics on Production Environment
• Interactive Analytics• Personalizing Dashboards • Customised Alerts
Summary
• Start small and scale as you grow• Minimum HA Deployment
– 2 Nodes
• Fully Distributed Deployment – 8+ Nodes – Scale based on need, horizontally and vertically
• Analyser, Indexer, Receiver, Realtime (With Apache Storm), Dashboard
In God we trust; all others must bring data
- William Edwards Deming -
Thank You