interactive visualization of streaming data powered by spark
TRANSCRIPT
![Page 1: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/1.jpg)
INTERACTIVE VISUALIZATION OF STREAMING DATA POWERED BY SPARK
![Page 2: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/2.jpg)
Streaming @Zoomdata
Visualizations react to new data delivered
Users start, stop, pause the stream
Users select a rolling window or pin a start time to capture cumulative metrics
![Page 3: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/3.jpg)
Drivers for Streaming Data
Data Freshness Time to Analytic Business Context
![Page 4: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/4.jpg)
Challenges• Time
• Frequency
• Retention
• Synchronization
• Order
• Updates
![Page 5: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/5.jpg)
Addressing streaming @Zoomdata
Historical Revised
Receive Data JMS Kafka
Manipulate Stream Single JVM in Memory Spark Streaming
Hold Data in Buffer MongoDB Pluggable
Interact with Data Custom Code Pluggable
![Page 6: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/6.jpg)
Technology Cast• The Stream - Kafka, Kinesis, JMS• Processing Fabric - Spark Streaming• Landing Area - MemSQL, Solr, Kudu, Others
![Page 7: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/7.jpg)
How it looks
![Page 8: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/8.jpg)
With the rest of the app
![Page 9: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/9.jpg)
Scale Out
![Page 10: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/10.jpg)
Let’s put it all together• Process fruit orders from all
over• Buyers make orders via an
API• API is SMS
– [fruit] [quantity] e.g., apple 10
![Page 11: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/11.jpg)
Demo
![Page 12: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/12.jpg)
Benefits
• Contextual Expressiveness with Streaming Data
• Independent scalability (scale-up, scale-around)
• Expressiveness powered by Spark -- using Windowing (dataframe API with stream)
• DR COOP, other Data management concerns
![Page 13: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/13.jpg)
Future Work• Cross stream synchronization & fusion
• On-demand scale out and resource management via Mesos
• Schema evolution
• More extensible landing strategies
![Page 15: Interactive Visualization of Streaming Data Powered by Spark](https://reader036.vdocuments.us/reader036/viewer/2022062316/587156191a28ab8e5b8b52f5/html5/thumbnails/15.jpg)
Thank You