real world cloud application security [email protected]
DESCRIPTION
Real World Cloud Application Security [email protected]. About Me. Director of Engineering @ Netflix Responsible for: Cloud app, product, infrastructure, ops security Previously: Led security team @ VMware Earlier, primarily security consulting at @stake, iSEC Partners. Netflix, Inc. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/1.jpg)
Real World Cloud Application [email protected]
![Page 2: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/2.jpg)
About Me
• Director of Engineering @ Netflix• Responsible for:– Cloud app, product, infrastructure, ops security
• Previously:– Led security team @ VMware – Earlier, primarily security consulting at @stake, iSEC
Partners
![Page 3: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/3.jpg)
Netflix, Inc.
“Netflix is the world’s leading Internet television network with more than 33 million members in 40
countries enjoying more than one billion hours of TV shows and movies per month, including original
series . . .”
Source: http://ir.netflix.com
![Page 4: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/4.jpg)
APPSEC CHALLENGES
![Page 5: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/5.jpg)
Lots of Good Advice
• BSIMM • Microsoft SDL • SAFECode
![Page 6: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/6.jpg)
But, what works?
Forrester Consulting, 12/10
![Page 7: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/7.jpg)
Especially, given phenomena such as DevOps, cloud, agile, and the unique characteristics of an organization?
![Page 8: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/8.jpg)
CLOUD @ NETFLIX
![Page 9: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/9.jpg)
Availability
![Page 10: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/10.jpg)
“Undifferentiated Heavy Lifting”
![Page 11: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/11.jpg)
Netflix Culture
“may well be the most important document ever to come out of the Valley.” Sheryl Sandberg, Facebook COO
![Page 12: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/12.jpg)
Scale and Usage Curve
![Page 13: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/13.jpg)
Netflix is now ~99% in the cloud
![Page 14: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/14.jpg)
On the way to the cloud . . . (architecture)
![Page 15: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/15.jpg)
On the way to the cloud . . . (organization)
(or NoOps, depending on definitions)
![Page 16: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/16.jpg)
DEPLOYING CODE
![Page 17: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/17.jpg)
A common graph @ Netflix
Lots of watching in prime time Not as much in early morning
Old way - pay and provision for peak, 24/7/365
Multiply this pattern across the dozens of apps that comprise the Netflix streaming service
Weekend afternoon ramp-up
![Page 18: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/18.jpg)
Solution: Load-Based Autoscaling
![Page 19: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/19.jpg)
Autoscaling
• Goals:– # of systems matches load requirements– Load per server is constant– Happens without intervention (the ‘auto’ in autoscaling)
• Results:– Clusters continuously add & remove nodes– New nodes must mirror existing
![Page 20: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/20.jpg)
Every change requires a new cluster push(not an incremental change to existing systems)
![Page 21: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/21.jpg)
Deploying code must be easy(it is)
![Page 22: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/22.jpg)
Netflix Deployment Pipeline
Perforce/Git
Code changeConfig change
YUM
RPM withapp-specific
bits
Bakery/Aminator
Base image +RPM
AMI
VM template ready to launch
ASG
Cluster configRunning systems
![Page 23: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/23.jpg)
Operational Impact
• No changes to running systems• No systems mgmt infrastructure (Puppet,
Chef, etc.)• Fewer logins to prod• No snowflakes• Trivial “rollback”
![Page 24: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/24.jpg)
Security Impact
• Need to think differently on:– Vulnerability management– Patch management– User activity monitoring– File integrity monitoring– Forensic investigations
![Page 25: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/25.jpg)
Architecture, organization, deploymentare all different.
What about security?
![Page 26: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/26.jpg)
We’ve adapted too.Some principles we’ve found useful.
![Page 27: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/27.jpg)
POINTS OF EMPHASIS
![Page 28: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/28.jpg)
Points of Emphasis
• Integrate• Make the right way easy• Self-service, with exceptions• Trust, but verify
• Two contexts:1. Integration with your
engineering ecosystem2. Integration of your
security controls• Organization• SCM, build and release• Monitoring and alerting
28
![Page 29: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/29.jpg)
Integration: Base AMI Testing
• Base AMI – VM/instance template used for all cloud systems– Average instance age = ~24 days (one-time sample)
• The base AMI is managed like other packages, via P4, Jenkins, etc.• We watch the SCM directory & kick off testing when it changes• Launch an instance of the AMI, perform vuln scan and other checks
SCAN COMPLETED ALERT
Site name: AMI1
Stopped by: N/A
Total Scan Time: 4 minutes 46 seconds
Critical Vulnerabilities: 5Severe Vulnerabilities: 4Moderate Vulnerabilities: 4
![Page 30: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/30.jpg)
Integration: Control Packaging and Installation
•From the RPM spec file of a webserver:Requires: ossec cloudpassage nflx-base-harden hyperguard-enforcer
Pulls in the following RPMs: HIDS agent Config assessment/firewall agent Host hardening package WAF
![Page 31: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/31.jpg)
Integration: Timeline (Chronos)
•What IP addresses have been blacklisted by the WAF in the last few weeks?
• GET /api/v1/event?timelines=type:blacklist&start=20130125000000000
•Which security groups have changed today?• GET /api/v1/event?timelines=type:securitygroup&start=20130206000000000
![Page 32: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/32.jpg)
Integration: Static Analysis
•Available self-service through build environment –FindBugs, PMD
• Jenkins plugin to display graphs and support drill through to results
![Page 33: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/33.jpg)
Integration: Static Analysis
![Page 34: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/34.jpg)
Points of Emphasis
• Integrate• Make the right way easy• Self-service, with exceptions• Trust, but verify
• Developers are lazy
![Page 35: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/35.jpg)
Making it Easy: Cryptex
• Crypto: DDIY (“Don’t Do It Yourself”)• Many uses of crypto in web/distributed systems:– Encrypt/decrypt (cookies, data, etc.)– Sign/verify (URLs, data, etc.)
• Netflix also uses heavily for device activation, DRM playback, etc.
![Page 36: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/36.jpg)
Making it Easy: Cryptex
• Multi-layer crypto system (HSM basis, scale out layer)– Easy to use– Key management handled transparently– Access control and auditable operations
![Page 37: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/37.jpg)
Making it Easy: Cloud-Based SSO
• In the AWS cloud, access to data center services is problematic– Examples: AD, LDAP, DNS
• But, many cloud-based systems require authN, authZ– Examples: Dashboards, admin UIs
• Asking developers to securely handle/accept credentials is also problematic
![Page 38: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/38.jpg)
Making it Easy: Cloud-Based SSO
• Solution: Leverage OneLogin SaaS SSO (SAML) used by IT for enterprise apps (e.g. Workday, Google Apps)
• Provides a single & centralized login page• Built base module to make SSO/authN trivial
![Page 39: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/39.jpg)
Points of Emphasis
• Integrate• Make the right way easy• Self-service, with
exceptions• Trust, but verify
• Self-service is perhaps the most transformative cloud characteristic
• Failing to adopt this for security controls will lead to friction
![Page 40: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/40.jpg)
Self-Service: Security Groups
• Asgard cloud orchestration tool allows developers to configure their own firewall rules
• Limited to same AWS account, no IP-based rules
![Page 41: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/41.jpg)
Points of Emphasis
• Integrate• Make the right way easy• Self-service, with exceptions• Trust, but verify
• Culture precludes traditional “command and control” approach
• Organizational desire for agile, DevOps, CI/CD blur traditional security engagement touchpoints
![Page 42: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/42.jpg)
Trust but Verify: Security Monkey
• Cloud APIs make verification and analysis of configuration and running state simpler
• Security Monkey created as the framework for this analysis
• Includes:– Certificate checking– Firewall analysis– IAM entity analysis– Limit warnings– Resource policy analysis
![Page 43: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/43.jpg)
Trust but Verify: Security Monkey
From: Security MonkeyDate: Wed, 24 Oct 2012 17:08:18 +0000To: Security AlertsSubject: prod Changes Detected
Table of Contents: Security Groups Changed Security Group <sgname> (eu-west-1 / prod) <#Security Group/<sgname> (eu-west-1 / prod)>
![Page 44: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/44.jpg)
Trust but Verify: Exploit Monkey
• AWS Autoscaling group is unit of deployment, so changes signal a good time to rerun dynamic scans
On 10/23/12 12:35 PM, Exploit Monkey wrote:
I noticed that testapp-live has changed current ASG name from testapp-live-v001 to testapp-live-v002.
I'm starting a vulnerability scan against test app from these private/public IPs:10.29.24.174
![Page 45: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/45.jpg)
Takeaways
• Netflix runs a large, dynamic service in AWS
• Newer concepts like cloud & DevOps need an updated approach to application security
• Specific context can help jumpstart a pragmatic and effective security program
• Don’t swim upstream - integrate and collaborate with your engineering partners
![Page 46: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/46.jpg)
Netflix References
• http://netflix.github.com• http://techblog.netflix.com• http://slideshare.net/netflix
![Page 47: Real World Cloud Application Security chan@netflix.com](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681685e550346895ddea120/html5/thumbnails/47.jpg)
Other References
• http://www.webpronews.com/netflix-outage-angers-customers-2008-08• http://www.pcmag.com/article2/0,2817,2395372,00.asp• http://www.readwriteweb.com/archives/etech_amazon_cto_aws.php• http://bsimm.com/online/• http://www.microsoft.com/en-us/download/confirmation.aspx?id=29884• http://www.slideshare.net/reed2001/culture-1798664• http://techcrunch.com/2013/01/31/read-what-facebooks-sandberg-calls-
maybe-the-most-important-document-ever-to-come-out-of-the-valley/• http://www.gauntlt.org