devops at obama for america(2012) and the dnc (devops days nyc jan 2013)
DESCRIPTION
Leo Zhandovsky (@leozh, formerly DNC, now AWS Solutions Architect) and I gave a brief talk about DevOps at the DNC and OFA campaign for DevOps Days in New York City. Video (Starts ~1hr55min) - http://new.livestream.com/devopsdaysorg/nyc2013TRANSCRIPT
DevOps @ DNC & OFADevOps @ DNC & OFA
Leo Zhadanovsky@leozhDirector of Systems Engineering @Democratic National Committee
John Schneider@jdotpFormerly Obama for AmericaCurrently DevOps / Internet Jedi @ Mozilla Foundation
DevOps @ DNC - 2009 - 2011
Year 1● Cleanup● Virtualization● Documentation● Version Control● Making everything HA● Replacing all our network gear● Automation
Labs Development Team
● Created first versions of Call Tool, Voter Registration Site, Canvassing iPhone App
Labs Development Team
Start of DevOps Practices at the DNC● Agile Development● Usage of EC2● Continuous Integration● Deploys using Capistrano● Scaling up using custom AMIs● Monitoring using Nagios, Cacti, NewRelic,
Graphite, Pingdom, StatsD
Analytics Team
● Lessons learned from 2010 used in 2012● Big Data● Hardware vs AWS● Limitations of AWS at the time● What would we do differently now?● Developed software for data analysts,
modelers and field staff
Cloud (AWS)
● Unpredictable traffic patterns: No problem! (We had no clue those birther mugs would sell so well.)
● Makes entire org far more agile
● Apples to apples testing envs are a snap
● Cloud rules changes the way we manage systems
Dev + Ops = <3The New Mentality of Devops
SOA / 12 Factor
● Allow for easy migration, deployment● All components independently scaled
without retooling or re-architecting● Critical for building scalable, high-availability
apps in the cloud.● Goes well with Route53.
Dev + Ops = <3The New Mentality of Devops
Agile and Continuous● Tool agnostic (we used Pivatol, Basecamp
Classic, Basecamp Next, spreadsheets, github issues, zendesk tickets, etc)
● Small and fast releases. (we pushed ~40 new versions per day in the last weekend)
● SHIPIT! Build 'good enough' tools and iterate.
Dev + Ops = <3The New Mentality of Devops
Instrument Everything (and trust the data)
● CPU, traffic, response time, etc
● User/System events
● Deployment-related change metrics
● Logs are data too!
Dev + Ops = <3The New Mentality of Devops
Visible Ops (aka, how to turn PMs, Devs, Management into Ops partners)
● They all get Nagios, NewRelic, and Pagerduty accounts and training
● Activity feed of commits, deploys, alerts, changes.
● Cross-pollinate team stands, retros● Give everyone contextual data
Who doesn't love color charts?
Load Testing with NewRelic Deploys
Dev + Ops = <3The New Mentality of Devops
Fail Early, Fail Fast, Fail Well
● It is not if, but when. ● Learn how to detect & mitigate, and then
iterate.● Reduce the time to detect, the time to
resolve. ● Preventing all failure is not the DevOps role,
making failure hurt less or not hurt at all is DevOps role
Yo, mind if we bring the donations db down during the VP debate?
Oh Hai East Virginia Datacenter!
Other AWS Tips-Use EBS-backed instances, AMI-based deploys, and provisioned IOPS. -Use SQS. -Multi-AZ, Multi-Region early-Autoscale from any metric, push your own metrics to Cloudwatch even-Setup ifttt or zapier alerts on AWS status pages-Check out billing broken out by tag, and TrustedAdvisor
S3 microsites are awesome
Other Things That Helped
New Relic (We love you New Relic)Asgard (Thanks Netflix)StatsD/Graphite and CubismLogglyBeesWithMachineGunsAkamai DSA and GTMBotoPalominoDB is awesome!Lots of caffeineLastly....culture, culture, culture.
Election Day