vayacondios: divine into complex systems
DESCRIPTION
This presentation given by Flip Kromer and Huston Hoburg on March 24, 2014 at the MongoDB Meetup in Austin. Vayacondios is a system we're building at Infochimps to gather metrics on highly complex systems and help humans make sense of their operation. You can think of it as a "data goes in, the right thing happens" machine: send in facts from anywhere about anything, and Vayacondios will promptly process and syndicate them to all consumers. Producers don't have to (or get to) worry about the needs of those who will use the data, or the details of transport, storage, filtering or anything else: the data will go where it needs to go. Each consumer, meanwhile, finds that everything they need to know is available to them, on the fly or on demand, without crufty adapters or extraneous dependencies. They don't have to (or get to) worry about the distribution of their sources, the tempo of update, or how the data came to be. Vayacondios was built for our technical ops team to monitor all the databases and systems they superintend, but it suggests a better way to build database driven applications of any kind. The quiet tyranny of developing against a traditional database has left us with many bad habits: not duplicating data, using models that serve the query engine not the user, assembling application objects from raw parts on every page refresh. Combining streaming data processing systems with distributed datastores like MongoDB let you do your query on the way _in_ to the database -- any number of queries, decoupled, of any complexity or tempo. The resulting approach is simpler, fault-tolerant, and scales in terms of machines and developers. Most importantly, your data models are purely faithful to the needs of your application, uncontaminated by differing opinions of other consumers or by incidentals of the robots that gather and process and store the data.TRANSCRIPT
![Page 1: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/1.jpg)
Vayacondios: Divine into Complex Systems
Huston Hoburg & Flip Kromer Infochimps, a CSC Company
MongoDB Austin 2014 March 24th
![Page 2: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/2.jpg)
Infochimps• Big Data Platform for Large Companies
• Cloud::Queries (ElasticSearch, MongoDB, HBase)
• Cloud::Hadoop (Dynamic Hadoop)
• Cloud::Streams (Storm+Trident)
• Managed Service, Enterprise Features
• Recently sold to CSC, and it’s quite awesome
• We’re Hiring (natch)
![Page 3: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/3.jpg)
Vayacondios
• Built for our Visibility Stack…
• … but we think it has wider use
!
• “Data Goes In, the Right Thing Happens”
• Prompt, Comprehensive and Faithful
![Page 4: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/4.jpg)
![Page 5: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/5.jpg)
![Page 6: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/6.jpg)
Circulatory
Immune
Clotting
![Page 7: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/7.jpg)
OK, Glass
“OK Glass, Show me a skeuomorphism”
![Page 8: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/8.jpg)
Immune
Circulatory
Digestive
Respiratory
![Page 9: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/9.jpg)
Non-Numeric Metrics
![Page 10: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/10.jpg)
Target INR = 2-3
Low Platelets = H.I.T. (bad)
Heparin (Blood Thinner)
![Page 11: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/11.jpg)
Low Platelets
• Folic Acid, Vitamin B12
• Medication (Valproic Acid, Singulair, Heparin)
• Sepsis
• HIV
• (about three dozen others)
![Page 12: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/12.jpg)
Systems• Anatomical Systems: Circulatory, Immune, etc
• Interventions: Drugs, Surgeries, …
• Course of Treatment: topline progress indicators
• Diagnosis
• Practitioner
• Medical Devices
![Page 13: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/13.jpg)
ICU
• Model the patient, not the data source
• Highlight Interactions among systems
• Highlight Interactions among numbers
• Broaden your view of “systems”
![Page 14: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/14.jpg)
Monitoring Sucks
![Page 15: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/15.jpg)
Operations
![Page 16: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/16.jpg)
System != Machine• Whole-System MongoDB:
• Machines it runs on, Volumes it uses
• Systems writing to it
• Applications and Collections
• Data Files, Logs, Repl Sets, Oplog, Arbiters
• Codebase repo, Cookbooks, Configuration
• Issue Tracker Tickets, Change Events
![Page 17: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/17.jpg)
Operations• Cognitive model for Humans, not from Robots
• Go beyond the Time-series Graph
• Highlight Interactions
• Link to Systems that write to this DB
• Link to Github for Repos & Cookbooks
• Drill into System
• Issues in Issue Tracker
• Broaden your view of “systems”
![Page 18: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/18.jpg)
• 15 clients, 15 architectures
• < 1 operator per client, 2 continents
• 1500 machines in 150 clusters
• 30+ technologies (HBase, MongoDB, Storm, …)
• 4 Providers (AWS, Metal, VCE, OpenStack)
• 3 Virtualizations (AWS, VMWare, OpenStack)
• Max 21 minutes downtime / month (99.95% SLA)
Our Challenge
![Page 19: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/19.jpg)
Systems to Instrument• WholeSystems: ZookeeperSystem, ElasticsearchSystem, HbaseSystem, HadoopMapredSystem, HadoopHdfsSystem,
KafkaSystem, MysqlSystem, MysqlClientSystem, ListenerSetSystem, StormTridentSystem, MongodbSystem, NfsSystem, VayacondiosSystem, TachyonSystem, SplunkSystem, S3System, RdsSystem, PigSystem, HiveSystem, HueSystem
• Machines: ZookeeperMachine, ElasticsearchDatanodeMachine, HBaseRegionserverMachine, HBaseMasterMachine, HadoopDnttMachine, HadoopTtonlyMachine, HadoopNamenodeMachine, HadoopJobtrackerMachine, HadoopSecondaryNamenodeMachine, HadoopFailoverMonitorMachine, MysqlServerMachine, KafkaBrokerMachine, PlatformListenerMachine, StormBolterMachine, StormMasterMachine, MongodbMachine, NfsServerMachine, VayacondiosServerMachine, PlatformApiMachine, TachyonServerMachine, HueMachine
• Daemons: n, ElasticsearchDaemon, HbaseRegionserverDaemon, HbaseMasterDaemon, HadoopDatanodeDaemon, HadoopTasktrackerDaemon, HadoopNamenodeDaemon, HadoopJobtrackerDaemon, HadoopSecondaryNamenodeDaemon, HadoopFailoverDaemon, KafkaBrokerDaemon, MysqlDaemon, PlatformListenerDaemon, StormNimbusDaemon, StormUiDaemon, StormSupervisorDaemon, MongodbDatanodeDaemon, NfsServerDaemon, NtpDaemon, NfsClientDaemon, VayacondiosServerDaemon, TachyonServerDaemon, PlatformApiServerDaemon, HueBeeswaxDaemon
• Providers: AwsProvider, CloudTrailProvider, OpenstackProvider, VceProvider, ChefServerProvider, Route53Provider, ElbProvider
• Manifests: most of the above have a planned version and the realized version • Events: MachineLifecycle, CronJobLifecycle, ChefClientLifecycle • Build Artifacts:: FitDeployArtifact, DebArtifact, RpmArtifact, GemArtifact, AmiArtifact, OpenstackImageArtifact,
VceTemplateArtifact, NpmArtifact, TarballArtifact • PlatformApps: HadoopJobLifecycle (Hive, Pig, Wukong), TridentJobLifecycle, MountweaselLifecycle • OpsProcesses: IncidentLifecycle, ChangeRequestLifecycle, FiredrillLifecycle, GitCommitLifecycle, ProblemLifecycle (JIRA),
LunchladyLifecycle
![Page 20: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/20.jpg)
Vayacondios
• Visibility Stack for our operations team
• Open-sourcing this summer
• Internals in Ruby
• Access anywhere (HTTP or log file)
• MongoDB (but now please forget that fact)
![Page 21: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/21.jpg)
![Page 22: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/22.jpg)
Cognitive Model• MongoDB:
• is_a Data store
• has_many Network Services
• has_many Daemons
• has_many Machines
• has_many Volumes
• has_many Collections
• …etc
![Page 23: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/23.jpg)
Model DSL (domain-specific language)
![Page 24: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/24.jpg)
Model DSL (domain-specific language)
![Page 25: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/25.jpg)
Faithful• Whiteboard rule: how do folks talk about system?
• If you need it, it’s in the system
Prompt• As fast as joint laws of Economics & Physics allow
Comprehensive
![Page 26: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/26.jpg)
![Page 27: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/27.jpg)
![Page 28: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/28.jpg)
Biographizing Isn’t Pretty
![Page 29: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/29.jpg)
![Page 30: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/30.jpg)
Faithful to Source
• crap data => well-formed data
• uniform JSON-ready hash
• syntax cleaned up
• semantically unchanged
• encouraged to model it, but let Wookiee win
![Page 31: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/31.jpg)
![Page 32: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/32.jpg)
Write Contract
• Vaya Con Dios, “Go with God”. As the kingdom of heaven is unknowable, so is further fate of data:
• How used
• By Whom
• How Processed
• Where Stored
![Page 33: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/33.jpg)
![Page 34: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/34.jpg)
Reporters/Reports
• Assemble Biographies into Reports
• Faithful to application
• Don’t know when will be run, why, etc
![Page 35: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/35.jpg)
Presentation
![Page 36: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/36.jpg)
Dashboarding
![Page 37: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/37.jpg)
text metrictext metric
text metric text metrictext metric text metric text metric
text metric
Model-Driven Templates
![Page 38: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/38.jpg)
Repeatable Partials
![Page 39: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/39.jpg)
Model/Presenter/View
• Report == Model
• Reporter == Presenter
• Dashboard .xml == View
![Page 40: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/40.jpg)
Model/Presenter/View
• More targets that just dashboard!
• Splunk+PagerDuty Alerts
• Cucumber tests
• Auditing reports (Security, Good Manners)
![Page 41: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/41.jpg)
System Checks
• Correctness, Consistency
• Attached Directly to the Model
• No worthwhile distinction between QA (integration tests) and live Alerts
• Drive Splunk+Pager Duty for Alerts
• Author Cucumber specs(!) for QA tests
![Page 42: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/42.jpg)
Safe Systems
![Page 43: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/43.jpg)
System Drift
• Cognitive Model
• Discoverable Interface
• Testable Contract
![Page 44: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/44.jpg)
Inevitability• If configured and reported, consistency checks
• If reported, dashboard exists
• If is_a generic system (eg filesystem), gets correctness tests (eg “capacity < 75%”)
• If system A discovers system B:
• dashboard has link from A to B
• connectivity & security checks from A to B
![Page 45: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/45.jpg)
Interaction
• Monitoring systems do a terrible job here
• Hard sources of failure:
• Drift conceived != realized
• Interaction unexpected consequences
• Change oops
![Page 46: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/46.jpg)
Application Design
![Page 47: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/47.jpg)
Application Design• Visibility into complex systems:
• Biography of raw parts (raw Model) => Reporter (Presenter) => Summary of Systems (View-ready Model)
• Database-driven Application • Model =>
Presenter =>View
![Page 48: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/48.jpg)
Simple Blog
![Page 49: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/49.jpg)
Blog: Views
Author Page
Post Page
Index Page
![Page 50: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/50.jpg)
Blog: ViewsAuthor Page
Post Page
Index Page
PostSynopsisReport
PostReport
UserReport
CommentReport
![Page 51: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/51.jpg)
“Query on the way In”!
• New/Updated Post: Update Post triggers…
• Update PostReport
• Update SynopsisReport
• Update UserReport
![Page 52: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/52.jpg)
“Query on the way In”!
• User fullname changes: Update User triggers…
• Update UserReport
• Update their SynopsisReports
• Update their PostReports
• Update their CommentReports
![Page 53: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/53.jpg)
Vayacondios Contract
![Page 54: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/54.jpg)
Faithful• Whiteboard rule: how do folks talk about system?
• If you need it, it’s in the system
Prompt• As fast as joint laws of Economics & Physics allow
Comprehensive
![Page 55: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/55.jpg)
Faithful• Single concern: subject of the biography
• look at what’s offered, look at what reports need
Prompt• Run as often as needed (not your concern)
Comprehensive
![Page 56: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/56.jpg)
Faithful• One Reporter per Application (*) & Topic
• USCE Method: Utiliz’n, Saturat’n, Connections, Errors
Prompt• Run as often as needed (not your concern)
Comprehensive
![Page 57: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/57.jpg)
Benefits
• Separation of concerns:
• Source complexity (API, parsing, translation)
• Timing
• Transport
• Individual Applications
• Reliability
![Page 58: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/58.jpg)
Benefits
• Separation of concerns: Source, Timing, Transport, Individual Applications, Reliability
• No external libraries in application
• Uniform access times
• Reduce risk from multiple-dependencies
![Page 59: Vayacondios: Divine into Complex Systems](https://reader034.vdocuments.us/reader034/viewer/2022051314/54b7167a4a7959286f8b461d/html5/thumbnails/59.jpg)
So What?
• There’s not much to it: shims and conventions
• VCD is not MongoDB
• just like MongoDB is not mmap tables
• Power through constraint: