securing devops

SecDevOpssecuring DevOps

Area4110.6.2016 Zürich

Aarno AukiaVSHN AG - The DevOps Company

10.6.2016 VSHN AG | http://vshn.ch 2

About me

● MSc Computer Science ETH

● Security Operations @ Google

● Co-Founder & CTO @ Atrila (Security Operations)

● Co-Founder & CTO @ VSHN (DevOps)

● Spare Time: Event Networks/WiFi at Area41

● @aarnoaukia

● http://about.me/aarno


Agenda

● DevOps ?

● Where is the security ?

● Customer example 1

● Customer example 2

● Discussion


DevOps?

● Collaboration: Development (Dev) and Operations (Ops)

● Bring agile software engineering methods to operations

– Automation: infrastructure as a code, versioning/rollback

– Testing: continuous integration/testing/deployment● Bring operations engineering experience to developers

– Scalability: independent microservices

– Production insight: monitoring/logging/metrics● Together: make the application's owner happier


Dev + Ops collaboration

● Bring together Developers & Operations

● Practice agile Operations Engineering

● Counter fear of change with (automated) testing

● Provide developer and development infrastructure

– Tools for developers, preferably self-served


Infrastructure as code

● Change from hand-groomed servers to Operations Engineering (from pets to cattle)

● Speed & reliability

● Versioning & rollback

● Prerequisite for self-service

– Give each developer a full stack

– No manual changes in production

– As many testing instances as needed


Infrastructure tools

● Packaging code & dependencies for atomic deployment/rollback

– Deb/rpm, Docker● Infrastructure state management (configuration mgmt)

– Puppet, Salt, Chef, Ansible● Continuous Integration/Testing/Deployment

– Jenkins/TravisCI/GitlabCI/Atlassian Bamboo● Self-Service

– Vagrant/Docker or through Continuous Deployment


Infrastructure testing

● Bring software engineering best practice to operations

● Large complex infrastructure (as code) → many moving parts

– Unit testing each module (webserver setup, database setup, cache setup, etc)

– Functional end-to-end testing of full stack (request to cache delivers content from database)

● Basically the same thing as production service monitoring but for each change


Infrastructure feedback

● Collect all logs in ELK (Elasticsearch, Logstash & Kibana)

– Let the developers search for prod error root cause

– No sudo/root access to production needed

– Added value: merged & indexed● Collect Server & Application Metrics

– Correlate with deployments & site traffic


Software Delivery Automation


Where is the security ?


Developers● Duh!

● Education, education, education

● Concept/architecture/code audits

● Use proven libraries

● ...


Configuration management● Declare target state

● Enforce state every x minutes, e.g. 15min

● Establish baseline system security

– Services enabled/disabled

– System (admin) users, groups, keys, hashes, sudoers

– AAA (AD/LDAP) for 'normal users'

– Host firewall (e.g. iptables)

– Installed software


Logging● Audit logging (who changed what when)

● Application/request log

● As WORM as feasible for the customer

– Generally read-only for 'normal users'

– Restricted admin access● ELK-Stack

– Transport, parsing, ingest: Logstash

– Storage & Indexing: Elasticsearch

– Querying & Dashboard: Kibana


Service Monitoring● Layer 7: HTTP, SMTP, etc

● Layer 6: SSL, certificates, protocols, ciphers, etc

● System parameters

● Updates

● Backup

● Tool: Icinga2


Backup● As WORM as feasible

– Restricted admin access, no access for customer staf

– Only new data can be pushed● Servers are enrolled automatically by configuration

management

– Enforcing the backup target will not be in the same location/infrastructure

● Data encrypted at source server using multiple keys

● Control connections use SSL/TLS

● Continuously monitored, regularly restore-tested


Version Management● Everything is in version management = GIT

– Customer code

– Configuration management code & config● Changes/commits feed into audit log

● Shared or dedicated service

– Shared: github.com, bitbucket.com, gitlab.com

– Dedicated: Atlassian Bitbucket, Gitlab● AAA through AD/LDAP

● Since all Devs have offline copies: no credentials in code !


Continuous integration● Trigger Build/Package/Test/Deploy on each commit

– Targets configurable per repository, branch, tag

– Manual 'promote' e.g. of production release

– Feed into audit log

– Store completed build/package artifacts● Artifactory● (private) docker registry● Deb/RPM repository

● Feed back status to Git-GUI, dashboard, monitoring

● AAA through AD/LDAP


Automated testing● All code is tested automatically

– Customer code

– Config management code & parameters● Testing depth depends on customer...

– Syntax, coding style (lint), static code analysis

– Unit tests● 'Does this module do what it is supposed to do?'

– Functional tests● 'Does the application behave correctly end-to-end?'● detect changes in nikto/sqlmap output ?


Databases & Backends● Growing list of 'standard software' needed as backends for

customer applications

– MySQL/MariaDB/Galera/MaxScale, PostgreSQL, Redis, MongoDB, RabbitMQ, Memcached, Solr, Elasticsearch, NFS/DRBD, Ceph

● All services automatically deployed by configuration management

● Provide each service with sane config, clustering, credential management, firewall config, backup config, monitoring config


Web & Application Servers● Growing list of application servers

– PHP, Python, Ruby, Java/Tomcat, Java/Wildfly, Java/Play, Coldfusion, Docker

● Provide each service with sane config, firewall config, backup config, monitoring config

● Provide backend credentials through environment variables

– http://12factor.net● Other standard components:

– Apache, Nginx, Varnish, mod_security, HAproxy, OpenVPN, iptables, pacemaker, keepalived

http://12factor.net/


Customer case 1● Server stack (Puppet)

– Nginx, Varnish

– PHP versions 5.6 and 7

– MySQL/MariaDB-Galera-Cluster

– Memcached/Redis/Solr/Elasticsearch● Application Deployment/update (Ansible/SSH)

● Bundle know-how (settings, tunings, etc.) in common module, override if necessary per customer through YAML-File in Git-Repository

● Docker image for local testing/developing


Case1


Customer case 2● OpenShift: PaaS Plattform as a Service

● Docker, Kubernetes (Google), Openshift (Redhat)

● 100% Opensource, enterprise support available

● Swiss public PaaS: appuio.ch

● EU/US public: AWS

● Dedicated/private available worldwide

– AWS

– Enterprise on-premises


OpenShift


About VSHN

● Swiss DevOps & Ops Company, 17 people in Zürich

● Building the tools and workflows for self-service

● Managing web applications in any cloud

– We are cloud-agnostic: we run on AWS, MSA, GCE, DO, Hetzner, OVH, SafeSwissCloud, Cloudscale, Exoscale and on any on-premises Enterprise private cloud

● We work for Amazee Labs, Liip, Mercedes Benz Switzerland,Migros, SaltCinema, SIX Group, Sherpany, Sobrado, Starticket, Suisa, Taskfleet, zurichopenair.ch, etc.

● How can we help YOU?


Thanks

● Questions ?

● We're hiring System and Software Engineers @vshn_ch !

● Get in touch with @aarnoaukia, @tobruzh or @vshnemanuel

securing devops

Presentations & Public Speaking