securing application deployments in ci/cd environments (updated slides:
TRANSCRIPT
![Page 1: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/1.jpg)
Securing Application Deployments in CI/CD Environments
![Page 2: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/2.jpg)
Hello!I am Binu RamakrishnanPrincipal Security Engineer at Yahoo
You can find me at:@securitysauce
![Page 3: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/3.jpg)
Overview◎CI/CD platform overview.
◎High level objectives.
◎Threat modeling.
◎Security patterns and best practices.
![Page 4: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/4.jpg)
CI/CD
![Page 5: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/5.jpg)
CI/CD Deployments◎Single-tenant◎Multi-tenant
![Page 6: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/6.jpg)
Single-tenantDedicated build environments.
![Page 7: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/7.jpg)
Multi-tenantShared build environment.
Driven by economies of scale.
![Page 8: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/8.jpg)
CI/CD in Nutshell
![Page 9: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/9.jpg)
Enterprise CI/CD
![Page 10: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/10.jpg)
High Level Security Objectives
![Page 11: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/11.jpg)
![Page 12: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/12.jpg)
Verifiable Chain of Trust
![Page 13: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/13.jpg)
No Elevated Privileges for Build Jobs
![Page 14: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/14.jpg)
Threat Modeling
![Page 15: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/15.jpg)
CI/CD Admin Exclusive access to CI/CD
platform
Actors
Developer User/customer of the
platform.
CI/CD Platform
DeveloperLimited admin access to
platform.
![Page 16: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/16.jpg)
Trust Boundaries.
https://www.flickr.com/photos/nasamarshall/14596371842
![Page 17: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/17.jpg)
Trust Boundaries
Web interface◎ UI
◎ REST APIs
Application
![Page 18: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/18.jpg)
Trust Boundaries
Isolation based on Docker containers or VMs.
Host
![Page 19: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/19.jpg)
Trust Boundaries
Segmentation
Network
![Page 20: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/20.jpg)
Attack SurfaceEntry Points
Exit Points
![Page 21: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/21.jpg)
Entry Points
◎ Build web interface.
◎ Commit and build notification handlers.
◎ Source and package dependencies.
![Page 22: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/22.jpg)
Internal Entry Points
◎ Build slave endpoint (to launch builds; SSH).
◎ Other control plane endpoints/REST APIs.
![Page 23: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/23.jpg)
Exit Points
◎ Deployable artifacts.
◎ Build notifications - eg emails, IRC and Hipchat
messages.
◎ Build console logs.
◎ Git repo update with build status and badges.
![Page 24: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/24.jpg)
Threats
![Page 25: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/25.jpg)
Common Threats
◎ User account compromise & insider threats.
○User and platform admins.
◎ Network Intrusion.
○CI/CD internal and external endpoints.
![Page 26: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/26.jpg)
Build Slave CompromiseLarge attack surface, spread across multiple networks (iphone, Android, Server app etc.)
How?◎ A network level compromise, exploiting a vulnerability in build slave.
◎ Jobs break out of build container.Impact◎ Access to production servers.
◎ Listen to the network, spoof identity and access unauthorized data.
![Page 27: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/27.jpg)
Shared/Non-ephemeral Keys◎ Locally stored long-lived keys to access protected
services.Two forms:
◉ Shared key
◉ Per-job keys (eg. OAuth tokens)
![Page 28: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/28.jpg)
SSH over Unrestricted ShellAllows arbitrary commands to get executed on a remote host.
![Page 29: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/29.jpg)
Building External CodeAn attacker can take this path to get into internal networks, either by adding backdoors or exploiting known vulnerabilities with open source software.
![Page 30: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/30.jpg)
Baseline security controls that are expected to be there in 2016
![Page 31: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/31.jpg)
Existing Security Controls◎ Authenticated endpoints.
◎ Web application security practices - CSRF, HTTPS,
HTTPOnly cookies, XSS protection etc.
◎ Source control: granular authorization for repos.
◎ Use KMS to manage secrets; no hardcoded secrets.
![Page 32: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/32.jpg)
Security Patterns for Risk Reduction
![Page 33: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/33.jpg)
Ephemeral Keys
https://www.flickr.com/photos/articnomad/241620406
![Page 34: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/34.jpg)
Stateless Auth Architecture
https://www.flickr.com/photos/18946008@N06/14551311971/
![Page 35: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/35.jpg)
◎ Events:
○ Commit trigger.
○ Manual trigger from build UI.
○ Automated/cron job.
○ Trigger a downstream job.
◎ Upstream service stores downstream service credentials (OAuth, Shared Keys etc.).
◎ Equal trust on all components in the pipeline
![Page 36: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/36.jpg)
![Page 37: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/37.jpg)
Audit Trails
https://www.flickr.com/photos/adriensifre/8403355648
![Page 38: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/38.jpg)
Container, ToolChainHardening
http://www.publicdomainfiles.com/show_file.php?id=13493588417214
![Page 39: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/39.jpg)
Network Segmentation
https://www.flickr.com/photos/bretagne-balades/15355029654
![Page 40: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/40.jpg)
Do not pull PII or other sensitive info to build machine.
Minimal Builds
![Page 41: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/41.jpg)
Few more...SSH: Use Restricted ShellsHeadless SSH access for automated deployment should use a restricted shell.
Roll Keys PeriodicallyEstablish a process to periodically roll trust anchor keys (and do it periodically).
Restrict Job Console LogsRestrict build job console logs only to authorized users
Enable 2FAAdmins must follow good security hygiene and use 2FA to access platform application and hosts
Prune Admin Access ListKeep admin list small for build systems and Git repo access
Vulnerability Patch MgmtMaintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure
![Page 42: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/42.jpg)
Acknowledgements◎ Christopher Harrell
◎ St John Johnson
◎ Mike Shema
◎ Jeremiah Wuenschel
![Page 43: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/43.jpg)
Let’s recap major conceptsEphemeral KeysEphemeral keys are the future. Service providers should start supporting ephemeral keys for authorization
Stateless Auth ArchitectureAugment the trust dependency of 1:1 relationship between the pipeline components with workflow job tokens.
Audit LogsA verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD.
Minimal BuildsAvoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum
Network SegmentationNetwork level Isolation of CI/CD machines from other machines
ToolChain HardeningBuild tools and Docker containers must be adequately hardened.
![Page 44: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/44.jpg)
Thanks!Any questions?You can find me at:@securitysauce
![Page 45: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/45.jpg)
Appendix-A
![Page 46: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/46.jpg)
Use◎ Source code◎ Pre-built packages
Risks◎ Targeted back doors◎ Vulnerabilities
Hard Problem: Securing Supply Chain
This threat is more to do with applications than the CI/CD platform itself.
Open source components constitute a large part of modern Internet based applications.
![Page 47: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/47.jpg)
Appendix-B
![Page 48: Securing Application Deployments in CI/CD Environments (Updated slides:](https://reader035.vdocuments.us/reader035/viewer/2022081521/587ae0411a28ab542b8b631b/html5/thumbnails/48.jpg)
Credits
Special thanks to all the people who made and released these awesome resources for free:◎ Presentation template by SlidesCarnival◎ Photographs by Unsplash & Death to the Stock Photo (
license)