RUGGED DEVOPS WILL HELP YOU BUILD UR CLOUDZ

by @wickett and @ernestmueller

OUTLINE

• Us, And Why You Care What We Say

• The Cloud, And How It Is Threatening You

• Rugged, And Its New Approach To Security

• DevOps, And How It Is Driving Collaborative Solutions

• Combining Cloud, Rugged, and DevOps To Solve The Problem

• How We Did Cloud Security With DevOps At NI

• Introducing RuggedDevOps Tool: Gauntlt

NI  CONFIDENTIAL

@wicke'Senior  DevOps  Engineer

CISSP,  GWAPT,  CCSK,  GSEC,  GCFW

james@wicke'.me

@RuggedDevOps

theagileadmin.com

@ernestmueller

DevOps Platform Manager and Release Manager,

Bazaarvoice

ernest.mueller@gmail.com

theagileadmin.com

WHAT IS THE CLOUD?

(ISP -> colo -> MSP) + virtualization + HPC + (AJAX + SOA -> REST APIs) =

IaaS

IDE/4GLs + (EAI -> SOA) + SaaS + IaaS = PaaS

((web site -> web app) -> ASP) + virtualization + fast ubiquitous

Internet + [RIA browsers && mobile] = SaaS

[IaaS | PaaS | SaaS ] + [ devops | open source | noSQL ] = cloud

THE GRAND UNIFIED THEORY

CLOUDINESS

• An outsourced managed service

• providing hosted computing or functionality

• delivered over the Internet

• offering extreme scalability

• by using dynamically provisioned, multitenant, virtualized systems, storage, and applications

• controlled via REST APIs

• and billed in a utility manner.

“Cloud? I’ve been doing that since 1988. It’s just the same old thing

with a new name." - Technohipster

Pretty new:multitenant

massively scalableelastic self provisioning

pay as you goResulting benefits:

agilityeconomy of scale

low initial investmentscalable cost/opex

resilienceeasy delivery

Not new:virtualizationoutsourcingintegrationinterwebz

I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical,

economic, and national security.

I am rugged, not because it is easy, but because it is necessary... and I am up for the challenge.

I recognize that my code will be used in

ways I cannot anticipate, in ways it

was not designed, and for longer than it was

ever intended.

RUGGEDIZATION THEORY

Building solutions to handle adversity will cause unintended, positive benefits that will provide value that would have been unrealized otherwise.

No Pain, No Gain

RUGGED-ITIESMaintainability

AvailabilitySurvivabilityDefensibility

SecurityLongevityPortabilityReliability

WHAT NEEDS TO HAPPEN

• Focus on real security. FUD doesn’t benefit anyone – figuring out how to “make it happen” – securely – benefits everyone.

• It’ll take time for compliance standards to get with the times – but don’t assume the cloud can’t be compliant – some of your auditors have actually heard of VMs and know what to do

• Organizations have to accept risk to reap rewards.

• Agile has taught orgs the collaborative approach is best

• Lean has taught orgs to experiment and iterate

source: Gene Kim, “When IT says No @SXSW 2012”

SECURITY SEES...

• They give advice that goes unheeded• Business decisions made w/o regard of risk• Irrelevancy in the organization• Constant bearer of bad news• Feels ignored by their peers (you know,

those devops guys)• Inequitable distribution of labor

TRADITIONAL SECURITY

THE CLOUD RESPONSE

THE SEPARATION MODEL

DEVOPS

SERVICE LIFECYCLE

ANTIPATTERN!

Deploying Software Manually

ANTIPATTERN!

Deploying to a Production-like Environment Only after Development is

Complete

ANTIPATTERN!

Manual Configuration

Management of Production

Environments

CONTINUOUS INTEGRATION

• Check In Regularly

• Create an automated and comprehensive test suite

• Keep build and test short and fast

• All tests must pass before moving on

• Never Go Home on a broken build

• Never comment out failing tests

CONFIGURATION MANAGEMENT

• Infrastructure as Code (IaC)• Model driven deployment• Version control everything• Know Your Environment if

you want to make it defensible

RUGGED DEVOPSBRIDGING SECURITY AND DEVOPS

DEVOPS (+SEC)

• Increased trend driven by agile development towards tight collaboration between developers and operations staff

• Be the “security buddy”

• Embed with projects, don’t be a seagull

• By understanding, be understood

• How secure are things usually when people and teams all work separately?

THE 6 R’S RUGGED DEVOPS• repeatable – no manual steps• reliable - no DoS here• reviewable – aka audit• rapid – fast to build, deploy, restore• resilient – automated reconfiguration • reduced - limited attack surface

APPLY RUGGED DEVOPS TO THE CLOUD

• Start with a Rugged DevOps team

• Use a lot of firewalls

• Scan your code

• Source to system

• Threat modeling

• Watch for changes

• Pen Testing

BUILD A RUGGED DEVOPS TEAM

PEOPLE, PROCESS, TECH

PEOPLE AND PROCESS

• Sit near the dev and ops team, better yet, put them all on the same team• Track security flaws or bugs in the same bug tracking

system• Automate whenever possible• Involve team with vendors• Measurement over time and clear communication

USE FIREWALLS...(A LOT OF THEM)

Web

DB

Middle Tier Middle Tier

LDAP

Firewall

Firewall

Firewall

DMZ 1

DMZ 2

DMZ 3

Traditional 3-Tier Web Architecture

Web Web

firewall

firewallfirewall

firewallfirewall

DB

Middle Tier Middle Tier

LDAP

DMZ x3

DMZ x2

DMZ x2

Cloud Firewalls and DMZfirewall firewall

WebWebWeb

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

RepeatableVerifiable

Prod/Dev/Test MatchingControlledAutomated

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

firewall

firewallfirewall

firewallfirewall

Web

DB

Middle Tier Middle Tier

LDAP

firewall

Web

firewall

Web

RUGGED BENEFITS

• Control and traffic whitelisting• Config management• Reproducible, automated and source controlled• No accidental data traversal across products or

dev/test/prod tiers• Dev and Test identical to Prod tier

SCANTHECODE

• Scans for OWASP Top Ten and more• Security Scanning as a Service• Static and Dynamic scanning• Integrated into development process• Dynamic and Static scanning

SOURCETO SYSTEM

AUTOMATED PROVISIONING - PIE

• Programmable Infrastructure Environment (PIE)• Code can be version controlled• Make Infrastructure as code• Defined once, deployed many times• Eliminate repetitive task and human errors• Rollback capability

• a framework to define, provision, monitor, and control cloud-based systems

• written in Java, uses SSH as transport, currently supports Amazon AWS (Linux and Windows) and Microsoft Azure

• takes an XML-based model from source control and creates a full running system

• to define, provision, monitor, and control cloud-based systems

THREATMODELME

THREAT MODELING

• Understanding the threat profile of a system• Provide a basis for secure design

and implementation• Discover vulnerabilities• Provide feedback for the

application security life cycle

p. 29 in Threat Modeling, Swiderski, Snyder

WATCHMY CHANGES

HOST INTRUSION DETECTION SYSTEM

• Watch the file system (using hashing and timestamps)–/etc/–/usr/bin–…

• Change control for applications• Alert on changes and anomalies• PIE watchdog

PENTESTING

PENETRATION TESTING

• Use external and internal penetration testing• White box testing vs. Black box testing• Look for automation opportunities

(ruby, python, …)

BUT WHAT ABOUT SECURITY TESTING IN MY

CONTINUOUS INTEGRATION SYSTEM?

PUT YOUR CODE THROUGH THE GAUNTLT

GAUNTLET, N. AN ATTACK FROM ALL SIDES

Your web app

w3af

fuzzers

nmap

nessus

sqlmapmetasploit

You

dirbustercustom attacks

GAUNTLT IS BUILT FOR

CONTINUOUS INTEGRATION

GAUNTLT IS

AN ALWAYS-ATTACKING ENVIRONMENT FOR

DEVELOPERS

WITH ATTACKS WRITTEN IN EASY-TO-READ LANGUAGE

ACCESSIBLE TO EVERYONE INVOLVED IN DEV, OPS, TESTING, SECURITY, ...

GAUNTLT INCLUDES

WHY GAUNTLT?

SECURITY DOMAIN KNOWLEDGE IS GENERALLY A MYSTERY TO DEV TEAMS

GAUNTLT ALLOWS DEV AND OPS AND SECURITY TO COMMUNICATE AND COLLABORATE

GAUNTLT JOINS:

THE PHILOSOPHY OF RUGGED SOFTWARE

&OUTSIDE-IN

TESTING

LETS LOOK INSIDE A COUPLE OF THESE FILES

@gauntlet @run

Feature: Run nmap against a target and pass the value of the hostname from the profile.xml.

Background: Given nmap is installed

Scenario: Verify server is available on standard web ports Given the hostname in the profile.xml When I run nmap against the hostname in the profile on ports 80,443 Then the output should contain: """ 80/tcp open http 443/tcp open https """

feature for nmap:nmap.feature

Given /^nmap is installed$/ do steps %{ When I run `which nmap` Then the output should contain: """ nmap """ } end

When /^I run nmap against the hostname in the profile on ports (\d+),(\d+)$/ do |arg2, arg3| steps %{ When I run `nmap \"#{@hostname}\" -p80,443` }end

step definition for nmap:nmap.rb

lets run gauntlt with the nmap.feature

against google.com

wickett$ gauntlt

@gauntlet @runFeature: Run nmap against a target and pass the value of the hostname from the profile.xml.

Background: # features/nmap/nmap.feature:5 Given nmap is installed # features/step_definitions/nmap.rb:2

Scenario: Verify server is available on standard web ports # features/nmap/nmap.feature:8 Given the hostname in the profile.xml # features/step_definitions/profile.rb:1 When I run nmap against the hostname in the profile on ports 8080,443 # features/step_definitions/nmap.rb:12 Then the output should contain: # aruba-0.4.11/lib/aruba/cucumber.rb:98 """ 8080/tcp open http 443/tcp open https """...

Failing Scenarios:cucumber features/nmap/nmap.feature:8 # Scenario: Verify server is available on standard web ports

1 scenario (1 failed)4 steps (1 failed, 3 passed)0m0.341s

running gauntlt with failing tests

wickett$ gauntlt

@gauntlet @runFeature: Run nmap against a target and pass the value of the hostname from the profile.xml.

Background: # features/nmap/nmap.feature:5 Given nmap is installed # features/step_definitions/nmap.rb:2

Scenario: Verify server is available on standard web ports # features/nmap/nmap.feature:8 Given the hostname in the profile.xml # features/step_definitions/profile.rb:1 When I run nmap against the hostname in the profile on ports 80,443 # features/step_definitions/nmap.rb:12 Then the output should contain: # aruba-0.4.11/lib/aruba/cucumber.rb:98 """ 80/tcp open http 443/tcp open https """

1 scenario (1 passed)4 steps (4 passed)0m1.117s

running gauntlt with passing tests

WALK VS. RUN

• gauntlt has two modes: walk and run

• meaning fast and slow or smoke and full

• This is done by labels in cucumber

• For each feature you will get to decide if it is a @walk or a @run test or both

SOME REALIZATIONS

• The core of gauntlt needs to provide a set of functionality that encourages contributors to write extensions for their pen testing tools

• A gauntlt DSL (Domain Specific Language) will arise with words like target, scan, attack, host...

• Smoke tests and validation vs. long running testing (nightly/weekly)

JOIN THE PARTY!!FORK GAUNTLT ON GITHUB

HTTPS://GITHUB.COM/THEGAUNTLET/GAUNTLT

CLOUD & SECURITY RESOURCES

• Book: Cloud Security and Privacy (Mather, Kumraswamy, Latif)

• Jericho Forum (collaboration.opengroup.org/jericho/)

• Amazon AWS Security Center (aws.amazon.com/security)

• Austin Cloud User Group (acug.cloudug.org)

• Cloud Security Alliance (cloudsecurityalliance.org)

• CSA Austin Chapter (austincloud.org)

• CSA Security Guidance for Critical Areas in Cloud Computing

• ENISA Cloud Computing Risk Assessment

@ERNESTMUELLER

@WICKETT

CONTACT US!