devops connect: josh corman and gene kim discuss devopssec

#RSAC

SESSION ID:

Gene Kim Joshua Corman

Rugged DevOps

Going Even Faster

With Software Supply Chains CTO Sonatype @joshcorman

Researcher and Author IT Revolution Press @RealGeneKim

@joshcorman @RealGeneKim

Session ID:

Gene Kim

Total time: 45 minutes 15 min: where we’ve been (levelset the tribe) Josh: 7m Gene: 13m


Session ID: Session Classifica0on:

Josh Corman, Gene Kim VERY ROUGH 1ST Draft

Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed…

CLD-106 Intermediate

@joshcorman @RealGeneKim 4 10/23/2013

@joshcorman

~ Marc Marc Andreessen 2011

@joshcorman @RealGeneKim 5


@joshcorman

Trade Offs Costs & Benefits


Beyond Heartbleed: OpenSSL in 2014 (31 in NIST’s NVD thru December)

7

§  CVE-2014-3470 6/5/2014 CVSS Severity: 4.3 MEDIUM ß SEIMENS * §  CVE-2014-0224 6/5/2014 CVSS Severity: 6.8 MEDIUM ß SEIMENS * §  CVE-2014-0221 6/5/2014 CVSS Severity: 4.3 MEDIUM §  CVE-2014-0195 6/5/2014 CVSS Severity: 6.8 MEDIUM §  CVE-2014-0198 5/6/2014 CVSS Severity: 4.3 MEDIUM ß SEIMENS * §  CVE-2013-7373 4/29/2014 CVSS Severity: 7.5 HIGH §  CVE-2014-2734 4/24/2014 CVSS Severity: 5.8 MEDIUM ** DISPUTED ** §  CVE-2014-0139 4/15/2014 CVSS Severity: 5.8 MEDIUM §  CVE-2010-5298 4/14/2014 CVSS Severity: 4.0 MEDIUM §  CVE-2014-0160 4/7/2014 CVSS Severity: 5.0 MEDIUM ß HeartBleed §  CVE-2014-0076 3/25/2014 CVSS Severity: 4.3 MEDIUM §  CVE-2014-0016 3/24/2014 CVSS Severity: 4.3 MEDIUM §  CVE-2014-0017 3/14/2014 CVSS Severity: 1.9 LOW §  CVE-2014-2234 3/5/2014 CVSS Severity: 6.4 MEDIUM §  CVE-2013-7295 1/17/2014 CVSS Severity: 4.0 MEDIUM §  CVE-2013-4353 1/8/2014 CVSS Severity: 4.3 MEDIUM §  CVE-2013-6450 1/1/2014 CVSS Severity: 5.8 MEDIUM §  …

As of today, internet scans by MassScan reveal 300,000 of original 600,000 remain unpatched or unpatchable


Heartbleed + (UnPatchable) Internet of Things == ___ ? In Our Bodies In Our Homes

In Our Infrastructure In Our Cars


Sarcsm: I’m shocked!

9


• The

The Cavalry isn’t coming… It falls to usıProblem Statement

Our society is adop0ng connected technology faster than we are able to secure it.

Mission Statement To ensure connected technologies with the poten0al to impact public safety and human life are worthy of our trust.

Collec9ng exis0ng research, researchers, and resources Connec9ng researchers with each other, industry, media, policy, and legal

Collabora9ng across a broad range of backgrounds, interests, and skillsets Catalyzing posi0ve ac0on sooner than it would have happened on its own

Why Trust, public safety, human life How Educa0on, outreach, research Who Infosec research community Who Global, grass roots ini0a0ve What Long-‐term vision for cyber safety

Medical Automo0ve Connected Home Public

Infrastructure

I Am The Cavalryı


The Rugged Manifesto

I am rugged... and more importantly, my code is rugged.

I recognize that software has become a foundation of our modern world.

I recognize the awesome responsibility that comes with

this foundational role.

I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer

than it was ever intended.

I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical,

economic, and national security.

I recognize these things - and I choose to be rugged.

I am rugged because I refuse to be a source of vulnerability or weakness.

I am rugged because I assure my code will support its

mission.

I am rugged because my code can face these challenges and persist in spite of them.

I am rugged, not because it is easy, but because it is

necessary... and I am up for the challenge.


Our Goals

§ Play Mad Chemists § The Best & Brightest of DevOps § The Best & Brightest of Security

§ Cause High Value / High Connection

§ Merge our Tribes for Mutual Awesomeness

§ Catalyze New Patterns and Solutions

#RSAC

SESSION ID:

Where We’ve Been

@RealGeneKim

The Downward Spiral…

@RealGeneKim

@RealGeneKim

IT Ops And Dev At War

19

@RealGeneKim

@RealGeneKim

There Is A Better Way…

@RealGeneKim

Google, Amazon, Netflix, Spotify, Etsy, Spotify, Twitter,

Facebook…

@RealGeneKim

10 deploys per day Dev & ops cooperation at Flickr

John Allspaw & Paul Hammond

Velocity 2009

Source: John Allspaw (@allspaw) and Paul Hammond (@ph)

@RealGeneKim

Little bit weird Sits closer to the boss

Thinks too hard

Pulls levers & turns knobs Easily excited Yells a lot in emergencies


@RealGeneKim

Ops who think like devs Devs who think like ops


@RealGeneKim

Dev and Ops


@RealGeneKim Source: Theo Schlossnagle (@postwait)

DevOps is incomplete,

is interpreted wrong, and is too isolated

@RealGeneKim

.*Ops

Source: Theo Schlossnagle (@postwait)

@RealGeneKim

^(?<dept>.+)Ops$

Source: Theo Schlossnagle (@postwait)

@RealGeneKim

Justin Collins, Neil Matatall & Alex Smolen from Twitter

*

@RealGeneKim

High Performers Are More Agile

30x

8,000x more frequent

deployments faster lead times than their peers

Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

@RealGeneKim

High Performers Are More Reliable

2x

12x the change

success rate faster mean time to recover (MTTR)

Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

@RealGeneKim

High Performers Win In The Marketplace

2x

50% more likely to

exceed profitability, market share & productivity goals

higher market capitalization growth over 3 years*

Source: Puppet Labs 2014 State Of DevOps

@RealGeneKim

Deploy Smaller Changes, More Frequently *

Source: http://www.facebook.com/note.php?note_id=14218138919

@RealGeneKim

“As a lifelong Ops practitioner, I know we need DevOps to make our work humane. In the past, I’ve worked every holiday, on my birthday, my spouse’s birthday, and even on the day my son was born.” Nathan Shimek Engineering Manager, New Context

@nathan_shimek

@RealGeneKim

The Three Ways

@RealGeneKim

The First Way: Outcomes §  Creating single repository for code and environments §  All Ops artifacts in version control §  Determinism in the release process §  Consistent Dev, Test and Production environments, all properly

built before deployment begins §  Developers checking in code daily, being productive §  Automated regression testing §  Features being deployed daily without catastrophic failures §  Decreased lead time §  Faster cycle time and release cadence

@RealGeneKim

The Second Way: Outcomes §  Peer review of code and environment changes

§  Disciplined automated testing enabling many simultaneous small, agile teams to work productively

§  Proactive monitoring of the production environment

§  Defects and security issues getting fixed faster than ever

§  High trust culture

§  All groups communicating and coordinating better

§  Everybody is getting more work done

@RealGeneKim

The Third Way: Outcomes *

#RSAC

SESSION ID:

Why It’s “Go Time”


Session ID:

Gene Kim

15 min: why we’re here, and why it’s “go time” Josh: 0m Gene: 7m


§ we’ve seen what true integration of infosec into the daily work of Dev and Ops; and it is good

§ key learnings of the DevOps Enterprise 2015

§ Ed Bellis example: Capital One: DevOpsSec

§ examples of practices: preventive, detective/corrective


New engineer to John Allspaw: “Is it okay for me to make this change?”

John Allspaw: “I don’t know. Is it?”


One Of The Highest Predictors Of Performance

Source: Typology Of Organizational Culture (Westrum, 2004)


DevOps Enterprise: Lessons Learned

§ On Oct 21-23, we held the DevOps Enterprise Summit, a conference for horses, by horses

§ Speakers included leaders from: § Macy’s, Disney, GE Capital, Blackboard, Telstra, US

Department of Homeland Security, CSG, Raytheon, Ticketmaster, Union Bank of California


Observations

§ They were using the same technical practices and getting the same sort of metrics as the unicorns § Target: 10+ deploys per day, < 10 incidents per month § Capital One: 100s of deploys per day, lead time of

minutes § Macy’s: 1,500 manual tests every 10 days, now 100Ks

automated tests run daily § Nationwide Insurance: Retirement Plans app (COBOL

on mainframe)


Observations

§ The transformation stories are among the most courageous I’ve ever heard – § Often the transformation leader was putting themselves

in personal jeopardy § Why? Absolute clarity and conviction that it was the

right thing for the organization

*


52 Source: Lean Enterprise (upcoming): Jez Humble, Joanne Molesky, and Barry O’Reilly

@RealGeneKim

Capital One: DevOpsSec

Source: Tapabrata Pal, Capital One

*


Heather Mickman, Target, Inc.

§ Abolished the TEP-LARB process

§ As a result, she won the Lifetime Achievement Award from her grateful team


What About Infosec?

§ Ed Bellis

§ Former CISO of Orbitz

§ VP Information Security at Bank of America

§ Currently CEO of Risk I/O


Risk I/O DevOps By the NumbersSmall & Frequent Commits• Average between 75 & 125

commits commits to Master/week• Simplicity is your friend




Security Automation at Risk I/OChef All the Things!

Test All the Things! (including security)Static + Dynamic ThroughoutContinuous Integration via CircleCI

Open-Sourced CookbooksModSecurity (airbag)

Nessus (air bag ctrl) Nmap (brakes)

SSH

iptables (shoulder belt)

encrypted volumes Duo 2FA openVPN

ChatOps = Slack + graphite + logstash + sensu + pagerduty




Security Automation at Risk I/OChef All the Things!

Test All the Things! (including security)Static + Dynamic ThroughoutContinuous Integration via CircleCI

Open-Sourced CookbooksModSecurity (airbag)

Nessus (air bag ctrl) Nmap (brakes)

SSH

iptables (shoulder belt)

encrypted volumes Duo 2FA openVPN

ChatOps = Slack + graphite + logstash + sensu + pagerduty

DevOps as a Compliance EnablerAutomation as Evidence & Doc

CookbooksLeveraging the ELK Stack

ElasticsearchLogstashKibana

Github + Code Climate + Risk I/OCompliance Automation Extra Credit: https://telekomlabs.github.io/

@Eellis

@RealGeneKim

The DevOps Audit Defense Toolkit h]p://bit.ly/DevOpsAudit

James DeLuccia IV Jeff Gallimore Gene Kim

Byron Miller

@RealGeneKim

Breaking The Bottlenecks In The Flow

§ Environment creation

§ Code deployment

§ Test setup and run (mention @rohansingh)

§ Overly tight architecture

§ Development

§ Product management

@RealGeneKim

@RealGeneKim

“deploys / day”

“deploys / day / dev”

#RSAC

SESSION ID:

Where We Want To Go


Session ID:

Gene Kim

15 min: where we want to go Gene: 0m Josh: 10m


§  outline concrete tangible things that can be done together to fulfill it

§  Accelerating to transition from here to there

§  Deming -> SW Supply Chain Rigor

§  Better/Fewer suppliers. §  Better Supply §  Traceability/Visibility throughout for Prompt/Agile recall §  “Congressional Bill” - now or never (Jim Routh)

§  Expanding the DevOps Enterprise community

§  we can have mutual benefit through DevOps and software supply chains

§  legislation


Innovate!

PRODUCTIVITY

TIME


Product Vulnerability Disclosures Following the HeartBleed Announcement (Circle Size Indicates CVSS Severity Score)

F5

New OpenSSL Disclosures (Both CVSS Level 10)Here

IBM

Cisco

IBM

McAfee

Initial 'HeartBleed' OpenSSL Disclosure (CVSS Level 5 (underscored))

Numb

er of

Prod

ucts

Includ

ed in

Ann

ounc

emen

t

0

10

20

30

40

50

60

70

80

90

100

110

120

Days Since HeartBeed Announcement0 10 20 30 40 50 60 70 80 90 100 110 120

X Axis: Time (Days) following ini0al HeartBleed disclosure and patch availability Y Axis: Number of products included in the vendor vulnerability disclosure Z Axis (circle size): Exposure as measured by the CVE CVSS score

COMMERCIAL RESPONSES TO OPENSSL


h]ps://www.usenix.org/system/files/login/ar0cles/15_geer_0.pdf

For the 41% 390 days CVSS 10s 224 days


True Costs & Least Cost Avoiders

ACME

Enterprise

Bank

Retail

Manufacturing

BioPharma

Educa0on

High Tech

Enterprise

Bank

Retail

Manufacturing

BioPharma

Educa0on

High Tech

Enterprise

Bank

Retail

Manufacturing

BioPharma

Educa0on

High Tech


ON TIME ON BUDGET ACCEPTABLE QUALITY/RISK


ON TIME. Faster builds. Fewer interrup9ons. More innova9on.

ON BUDGET. More efficient. More profitable. More compe99ve.

ACCEPTABLE QUALITY/RISK. Easier compliance. Higher quality. Built-‐in audit protec9on.





Agile / CI


DevOps





DevOps / CD

Agile / CI


SW Supply Chains





SW Supply Chain

DevOps / CD

Agile / CI


SW Supply Chains


Toyota Advantage

Toyota Prius

Chevy Volt

Unit Cost 61% $24,200 $39,900

Units Sold 13x 23,294 1,788

In-‐House Produc0on 50% 27% 54%

Plant Suppliers 16% (10x per) 125 800

Firm-‐Wide Suppliers 4% 224 5,500

Comparing the Prius and the Volt


H.R. 5793 “Cyber Supply Chain Management and Transparency Act of 2014”

§  Elegant Procurement Trio 1) Ingredients: §  Anything sold to $PROCURING_ENTITY must provide a Bill of

Materials of 3rd Party and Open Source Components (along with their Versions)

2) Hygiene & Avoidable Risk: §  …and cannot use known vulnerable components for which a

less vulnerable component is available (without a written and compelling justification accepted by $PROCURING_ENTITY)

3) Remediation: §  …and must be patchable/updateable – as new vulnerabilities will

inevitably be revealed

#RSAC

SESSION ID: Go Forth… …and be Rugged


@RuggedSoftware


Want More Learn More? To receive the following: §  A copy of this presentation §  The 140 page excerpt of The Phoenix Project §  Videos and slides from DevOps Enterprise 2014 §  Information on DevOps Enterprise 2015 §  Link to the DevOps Audit Defense Toolkit §  Announcement of The Phoenix Project audiobook §  See early drafts of our upcoming DevOps Cookbook Just pick up your phone, and send an email:

To: [email protected] Subject: devops

[email protected]

devops

devops connect: josh corman and gene kim discuss devopssec

Technology