monitoring @ scale over diverse data sources @ paypal - druid, tsdb, hadoop
TRANSCRIPT
![Page 1: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/1.jpg)
Monitoring at Scale over diverse data sources @ PayPal – Druid,
TSDB, Hadoop Grafanacon, December 1, 2016
Senthil Pandurangan - Platform Architect Abhishek Sant – Senior Software Engineer
![Page 2: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/2.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Agenda • Introduction
• Monitoring Platform Architecture
• Grafana Druid Plugin
• Q & A
![Page 3: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/3.jpg)
3
PayPal Command CenterPowered by Monitoring Team
![Page 4: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/4.jpg)
4
PayPal Monitoring Ecosystem - Sherlock
APIs
Logs Processing
Metric Processing Insights Engine
Alerts
Visualization
Events Stream
Metrics Stream
Topology
Logs Stream
Pools & Hosts
PaaS
IaaS
Network
Databases
CAL (Logging)
Remediation OrchestrationConfiguration
Sherlock is a unified monitoring, alerting and logging platform that exposes real time data and insights, enabled by an extensible visualization platform.
Developers
Analysts
Command Center
SRE
![Page 5: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/5.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Tools & Technologies
Storage
Processing
Visualization
![Page 6: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/6.jpg)
6Ingest Queue Process Store Query
PayPal Monitoring Platform - Architecture
Message Bus
TSDBApp
Logs
App
App
Services
DBLB
NO
SQ L
Met
ric s
Stream processing
Met
ric sLo
gs
Log Views
Hadoop Reports
Custom Visualizatio
ns
Alerts
![Page 7: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/7.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Agenda • Introduction
• Monitoring Platform Architecture
• Grafana Druid Plugin
• Q & A
![Page 8: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/8.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Grafana Druid Pluginhttps://grafana.net/plugins/abhisant-druid-datasource/installation
![Page 9: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/9.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Grafana - Adding Druid Data source
![Page 10: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/10.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Grafana Druid Plugin – Query Editor
![Page 11: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/11.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Druid Query Types
Time Series
Top N
Group by
Select
![Page 12: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/12.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Multiple Data sources
Time Series
Top N
Group by
Select
TSDB
Druid
Hadoop Reports
![Page 13: Monitoring @ scale over diverse data sources @ PayPal - Druid, TSDB, Hadoop](https://reader035.vdocuments.us/reader035/viewer/2022062223/587adda91a28ab542b8b5bf7/html5/thumbnails/13.jpg)
©2016 PayPal Inc. Confidential and proprietary.
Thank You