Securing Application Deployments inMulti-tenant CI/CD Environments

O’Reilly Security Conference NY 2016

HELLO!I am Binu Ramakrishnan

Principal Security Engineer at Yahoo

You can find me at:@securitysauce

github.com/prbinu

Overview

▹ A brief introduction to CI/CD platform

▹ Challenges with multi-tenant CI/CD environments

▹ Threat modeling

▹ Security patterns and best practices

CI/CD

A CI/CD pipeline is a set of jobs chained to achieve continuous delivery, and deployment of applications to production.

CI/CD DeploymentsSingle-tenant | Multi-tenant

Single-tenant

▹ Dedicated build environments

▹ Distributed

▹ Tenant and provider are the same

▹ Limited security impact

Multi-tenant▹ Shared build environments

▹ Driven by economies of scale

▹ Focused on operational efficiency

▹ Allow enterprise to centrally enforce dev stds & compliance controls

▹ Gateway to enterprise networks

Multi-tenant CI/CD - Security Risks

▹ This platform can be an attractive target for attackers because:

▸ Consolidation of large number of long-lived keys in one place

▸ It acts as a corridor that directly connects to large number of machines, bypassing firewalls

▹ A successful attack can have a significant impact to an enterprise

▸ Leads to exposure of user’s personal info, application credentials, TLS private keys etc.

Denial of Service (DoS) Attacks

* http://bit.ly/2exRpWm

“Enterprises relying on SaaS apps had no choice but to sit and wait and hope their providers got back online as soon as possible.” *

CI/CD in Nutshell

Enterprise CI/CD

Threat Modeling

CI/CD Admin Exclusive access to the CI/CD platform

Actors

User/Developer Person who uses the platform and run their build pipelines.

CI/CD Platform DeveloperLimited admin access to the

platform.

Trust Boundaries

▹ Application - web interface, REST APIs

▹ Host - isolation between host OS and the build job

▹ Network - segment build machines from other machines and from Internet

Entry Points

▹ Build web interface

▹ Commit handlers

▹ Internal endpoints

▸ Build slave endpoint (to launch builds; SSH)

▹ Deployable artifacts

▹ Build notifications - eg. emails, IRC and Hipchat messages

▹ Build console logs

.

Attack Surface

Exit Points

Threats

16

Common Threats

▹ User account compromise & insider threats▸ User and platform admins

▹ Network Intrusion▸ CI/CD internal and external endpoints

Local Storage of Non-ephemeral Keys

▹ Long-lived keys are stored locally to allow build jobs to access protected services.

▹ OAuth tokens, Shared secrets, SSH private keys etc.

Weak Security Controls

Build jobs require access to services as part of their execution (eg. artifacts store). Protection mechanism used by those services are less effective in following cases:

▸ IP whitelist based protection

▸ Shared secrets: shared (and accessible) by all build jobs

The above controls are susceptible to spoofing in multi-tenant environments.

Build Slave CompromiseLarge attack surface, spread across multiple networks (iphone, Android, Server app etc.)

How?▹ A network level compromise, exploiting a vulnerability on build slaves

▹ Jobs break out of build container

Impact▹ Listen to the network, spoof identity and access locally stored

credentials

▹ Gain access to unauthorized machines and steal sensitive data

Building External CodeAn attacker can take this path to get into internal network, either by adding backdoors or exploiting known vulnerabilities with open source software.

Security Patterns for Risk Reduction

Ephemeral Keys

https://www.flickr.com/photos/articnomad/241620406

JWT, OAuth, SSH-CA, AWS access key etc.

▹ Automated deployments require delegation

▹ Token Delegation: a controlled replay of tokens

▹ Ephemeral keys go well with token delegation

model

Capability Delegation & Stateless Auth

https://www.flickr.com/photos/18946008@N06/14551311971/

By passing capabilities (in the form of tokens) with build events, we eliminate the need to store long-lived keys in CI/CD platform

▹ Separate out auth components from the core build platform

▹ The Auth service intercepts and authenticates build events, and issues job tokens to downstream

build platform

▹ Stateless: Build platform no longer stores job specific long-lived keys

▹ Distributed: Per tenant Auth service for cloud based environments, hosted inside tenant’s network

▹ No consolidation of keys in one place

Audit Trails▹ Immutable and append-only

▹ Traceability: logs should capture build pipeline events end to end from commit to final deploy.

▹ Trail traits: who, how, when, sequence of events, start and end state.

▹ Audit trails produced by the Auth service can provide a verifiable chain of trust from commit to deploy.

Container & ToolChain Hardening

Network Segmentation

Colocating CI/CD platform with other corporate machines allow malicious build jobs to raid your internal network.

The recommendation is to zone CI/CD platform from other corporate systems.

Minimal Builds▹ Do not expose PII or other sensitive info to build machines.

▹ Do complex tests outside of build, in a separate environment you

control.

Few more...SSH: Use Restricted ShellsHeadless SSH access for automated deployment should use a restricted shell.

Roll Keys PeriodicallyEstablish a process to periodically roll trust anchor keys (and do it periodically).

Restrict Job Console LogsRestrict build job console logs only to authorized users

Enable 2FAAdmins must follow good security hygiene and use 2FA to access platform application and hosts.

Prune Admin Access ListKeep admin list small for build systems and Git repo access.

Vulnerability Patch MgmtMaintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure.

Acknowledgements▹ Christopher Harrell

▹ St John Johnson

▹ Mike Shema

▹ Jeremiah Wuenschel

Let’s recap major conceptsEphemeral KeysReplace long-lived keys with time-bound scoped tokens. Service providers should start supporting tokens for auth & authz.

Capability DelegationDelegate job capabilities with build events, and eliminate the need to store long-lived keys in CI/CD platform.

Audit TrailsA verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD.

Minimal BuildsAvoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum.

Network SegmentationNetwork level Isolation of CI/CD machines.

Build ToolChain HardeningBuild tools and Docker containers must be adequately hardened.

THANKS!Any questions?

You can find me at:@securitysaucegithub.com/prbinu

CREDITS

Special thanks to all the people who made and released these awesome resources for free:▹ Presentation template by SlidesCarnival▹ Photographs by Death to the Stock Photo

(license)