act 1: technologyblog.marc-seeger.de/assets/papers/2014_05_20_devops_boston_heartbleed.pdfboston...
TRANSCRIPT
![Page 1: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/1.jpg)
Marc Seeger (@rb2k)Boston Devops Meetup
May 20th 2014
at
![Page 2: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/2.jpg)
Act 1: Technology
![Page 3: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/3.jpg)
How it all started7:24 PM
![Page 4: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/4.jpg)
How it all started7:30 PM
![Page 5: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/5.jpg)
How it all started7:26 PM
![Page 6: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/6.jpg)
How it all started7:33 PM
![Page 7: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/7.jpg)
How it all started
![Page 8: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/8.jpg)
Quick risk assessment
Lucid: [00:35:27] [email protected]:~# openssl version OpenSSL 0.9.8k 25 Mar 2009 !Precise: [00:34:37] [email protected]:~# openssl version OpenSSL 1.0.1 14 Mar 2012
![Page 9: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/9.jpg)
Where’s Waldo OpenSSL8000 EC2 Machines: - 99.9% of them puppetized - Candidates:
- Balancers - SVN Servers - Appliances
- ELBs - 3rd party AMIs
- Unique little snowflakes(Jira, Crucible,…)
![Page 10: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/10.jpg)
Let the patching begin
![Page 11: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/11.jpg)
RolloutAustralia: !Con: - Spiders - Snakes
!Pro: - Ops is awake
![Page 12: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/12.jpg)
Rollout
![Page 13: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/13.jpg)
Scan
www
![Page 14: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/14.jpg)
Waiting on ELBs…
![Page 15: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/15.jpg)
Internal Certificates
![Page 16: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/16.jpg)
Suddenly: “reverse” Heartbleed
![Page 17: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/17.jpg)
Act 2: Communication
![Page 18: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/18.jpg)
Internal
• Pre-determined chat rooms
• Dial-in conference bridges
• A communication plan
Thanks SSAE-16, PCI and FedRAMP… I guess :)
![Page 19: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/19.jpg)
Statuspage + Twitter
* Powered by StatusPage.io
*
![Page 20: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/20.jpg)
Documentationhttps://docs.acquia.com/articles/heartbleed-acquia-cloud
![Page 21: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/21.jpg)
Proactive communication
Phone calls by Acquia support, TAMs, …
![Page 22: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/22.jpg)
Since then: Post mortem
![Page 23: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/23.jpg)
Since then: Incident Commander
(shamelessly stolen from Heroku)http://en.wikipedia.org/wiki/Incident_command_system
![Page 24: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/24.jpg)
Since then: Dedicated resource to vet security threats
![Page 25: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/25.jpg)
Since then: Clean up intranet docs
![Page 26: Act 1: Technologyblog.marc-seeger.de/assets/papers/2014_05_20_DevOps_Boston_Heartbleed.pdfBoston Devops Meetup ... 2014_05_20_DevOps_Boston_Heartbleed.key Created Date: 5/20/2014 9:18:13](https://reader034.vdocuments.us/reader034/viewer/2022042313/5edd1d6dad6a402d666815d3/html5/thumbnails/26.jpg)
Since then: Additional tooling