image management in a private cloud
TRANSCRIPT
Image Management in a Private Cloud
Igor Bolotin & Richard Gooch
Image Management is not just for politicians and celebrities
Environment• Existing applications run on machines with ad-hoc management and
a variety of Operating Systems:
– Various Linux flavours (RHEL, Ubuntu, CentOS) and versions
– Various MS Windows versions
• Desire to move applications to Cloud-based Virtual Machines: the first baby step
• The carrot: Cloud VMs are easier to provision and manage (for the users)
• We provide Cloud Platform Services (DB, LMM, LB, etc.)
• Longer term: move services to orchestrated containers
Who is Responsible?• Public cloud - typically guest OS is purely tenant responsibility.
– Cloud service provider delivers and manages base images
– Building customer images, vulnerability scanning, patching, upgrades are all handled by tenants
• Private cloud - responsibility for maintaining security is shared
– Custom images are must have for efficient operations (“it takes 40 minutes to install all my packages using base image”), but maintaining these images is overhead for developers
– When new vulnerabilities are discovered - all teams scramble to patch systems and images - not efficient
– Developers and production systems have different needs and development images could include tools that are not allowed in production
• OS images security and governance should be provided as a service to cloud users
• Everyone can upload and publish images (image sprawl, OS sprawl)
• No vulnerability scanning or patching approach - patchwork of
security practices per customer
• Developer VMs are never scanned or patched
• Publicly accessible systems are vulnerable to most basic threats
• Agility without security
Governance Pitfalls - Too Lax
Governance Pitfalls - Going Too Far
• Same few base images used for all users
• No custom images allowed
• No root or sudo allowed by default
• Same configuration management system (Puppet) used to manage
VMs for all customers
• Puppet overrides all unauthorized changes to the systems
(removing packages, replacing configuration files, etc.)
• Security without agility
Balanced Approach - Agility with Security
• Different types of images with clear ownership and responsibilities
• Role-based access control
• Capacity management for images and snapshots
• Managing images as code with reviews and continuous integration
• Vulnerability and patch management as a service
• Configuration management
Public Images - Official and Supported
• Provider base images
• Platform images - i.e. images used for cloud platform services
• Built and maintained by Symantec Cloud team
– Image build process is internally open-sourced
– Anyone can use same tools and scripts to build custom images
• Naming conventions
• Replace, archive, cleanup
Private and Community Images
• Private images
• Shared private images
• Community images - shared with community, without warranty
– https://wiki.openstack.org/wiki/Glance-v2-community-image-sharing
– https://review.openstack.org/#/c/124050/
• Not maintained/supported by Symantec Cloud team
– Failures might need to be reproduced using an official image in order to request support
• Image owner is responsible for patching/remediation of critical vulnerabilities
– Defined SLA for remediation
– Disabled if not fixed
Role-Based Access Control
• Self-service Dev Class of Service
– Every developer can build and upload image to their own project
– Sharing images with other tenants is allowed
• Upload and sharing images in production requires “project image admin” role
• Making image public or deleting public image requires “cloud image admin” role
Capacity Management
• Storage quota for images and snapshots
• Snapshots - 3 months expiration by default
• Archival and cleanup process
• Per-project policies
• “Keep this one forever”
Terminated Accounts
• Ownership transfer
– Take over or take down!
– Unclaimed - disabled and deleted after grace period
• Unclaimed & takedown notifications
– Immediate manager
– Project owner
– VM owners
Managing Images as Code• Continuous Integration for images
– All scripts for building images are stored in Git– Code changes are submitted for review (using Gerrit)– Images are built and tested automatically as part of the review process – Validation includes security/vulnerability scan– Base images are published to all Symantec Cloud regions after review and validation – Publishing process by default uses canary deployment in one region before rolling out
globally– Automated image update workflow takes care of hiding/archiving previous version
• Same CI/CD pipeline offered as a service for building custom images– Benefit - fixing images becomes a very simple process
Vulnerability Management and Patching
• Symantec Cloud provides tools and processes for image scanning and approval
• All publicly available (shared) images must pass security testing (hardened and scanned for
vulnerabilities)
– before image can be shared with other tenants in production class of service
– before image can be published as community image
– before image can be published as an official/supported public image
Vulnerability Management and Patching (cont.)
• All images must be periodically (re)scanned for vulnerabilities
• If critical vulnerabilities discovered - image is marked as “vulnerable”
• Information about image vulnerability status is reported via cloud API, CLI and UI
• Image owners and VM owners using a vulnerable image are notified proactively via email
• Latest vulnerability scan report for each image is available for review
• Summary results of each vulnerability scan are recorded for audit trail
Vulnerability Management and Patching (cont.)
• SLA is defined for fixing public/shared images
– Vulnerable images that are not fixed within defined time window are disabled automatically
– Image owners and customers using vulnerable images (have VMs launched from that image) are
notified:• about image marked as vulnerable (include list of impacted VMs, periodic reminder notifications)• about image remediation• about image takedown https://blueprints.launchpad.net/glance/+spec/deactivate-image
• VMs that were built from vulnerable images are marked as “potentially vulnerable” and should either be patched or replaced using fixed images
• Symantec Cloud team reserve the final right to terminate any system that presents a real, immediate security threat.
Configuration Management
• Base provider images are configuration management agnostic – cloud-init scripts take care of basic VM configuration - setting up user access and
networking
– choice of configuration management is not forced by Symantec Cloud team
• Platform images are “opinionated” and rely on a specific configuration management approach (puppet)
Q&ALet’s talk…
Image Management is not just for politicians and celebrities
