scaling graphite for application metrics
TRANSCRIPT
About CrowdStrike
Big Data Security Company
Focus on targeted, state sponsored attacks and attribution
Single enterprise can generate 2+TB of machine data per day
MicroService architecture w/1000’s of VMs running.
We use goodies like AWS, Kafka, Cassandra, Elastic Search, Hadoop, Scala, GoLang
What is Graphite?
• Captures Numeric, Time-Series Data
• Metric: test.myapp.host1.logins
• Value: 64
• Timestamp: 1432077015
Graphite
• Composed of 3 projects
• Carbon - collects and records metrics
• Whisper - Backend storage mechanism
• Graphite-Web - HTTP frontend for graphing API
• Written in Python
What metrics to track?• counters, latencies, error rates • business metrics: sales, order latency, abandoned carts • refactoring, how do you know you’ve succeeded • hadoop metrics via MetricFactory • logins, login failures
Libraries
• JAVA/Scala: dropwizard.github.io
• Go: github.com/rcrowley/go-metrics
Specs: AWS biased
NO MAGNETIC: SSD ONLY !
Load Balancer: HAProxy or ELBS
Cache/Whisper: i2.2XL (8 core)
1.6TB RAID0 SSD XFS filesystem
1 cache per core
Relays: c3.large (2 core)
in autoscale group give it CPU#1
taskset -apc 1 PID
Web Tier: M3 memory instances
Use MySQL or Postgres Memcache ON
Data Retention[garbage_collection]
pattern = garbageCollections$
retentions = 10s:7d,60s:90d
!
Data size is fixed, inserted with NULLS, real data overwrites so files don’t grow larger. Makes estimation easier.
Modern Way Consistent Hashing
• Less shuffling of data when adding or removing nodes
• Graphite utilizes this approach much like Cassandra’s mechanism for distributing data
Graphite Downside: !
unlike Cassandra, re-balancing data isn’t automatic. You’ll need Carbonate or BuckyTools
!
https://github.com/jssjr/carbonate or
https://github.com/jjneely/buckytools
Key Resourceshttps://gist.github.com/obfuscurity/63399584ea4d95f921e4 !
http://bitprophet.org/blog/2013/03/07/graphite/ !
ClusterServers vs Destinations https://answers.launchpad.net/graphite/+question/228472 !
Tuning for 3m writes a minute https://answers.launchpad.net/graphite/+question/178969 !
http://www.aosabook.org/en/graphite.html !
https://grey-boundary.io/the-architecture-of-clustering-graphite/ !
WE’RE HIRING :)
•Offices in Irvine / Seattle / DC •Massive Scale •Fast Growing Company •Distributed Systems
An Environment Made For Engineers
•Open Source Friendly •Stock / Bonus plans •GeekDesks
[email protected] twitter: @jimplush
crowdstrike.com/careers
The Tech For All-The-Things!