creating a serverless ami certification pipeline

Creating a Serverless AMI Certification PipelineSEATTLE AWS ARCHITECTS & ENGINEERS MEETUPDAVID F. SEVERSKI - JANUARY 31, 2017

2

Background I’m in information security

Most of my time is in the risk space

Started trying to figure out this cloud thing 5 years ago

AWS Full 5 Certified

Drinker of the DevOps Kool-Aid

3

GenesisJim Fink’s November AWS Inspector Talk

4

Lambda Already in Use AMI S3 Cleaner

Controlling AWS ML Endpoints

SNS-Slack Bridge

Periodic RDS Extractions

5

Enter Step Functions and Lambda Environment VariablesAnnounced at re:invent 2016

Step Functions and Environment Variables?!

6

Real World Architecture“Sometimes, it’s messy”

Image © Valve Software

7

Materials Will Be ProvidedGitHub, Slideshare, etc.

8

Problem StatementHow can I ensure that my AMIs are built without known vulnerabilities?

No persistent hosts

Rapidly changing images (Packer built AMIs)

Building a known good image verification system without statically running hosts (Jenkins, etc.)

9

Quick Lambda RefresherFunctions deployed without servers (that you manage anyways)

Maximum runtime of five minutes

Auto-scaled by AWS

Super cheap to run

Choice of languagesUsing Python 2.6 and Boto for this example

Can use IAM roles to access other AWS services

Wide support for triggering via other AWS Services

10

Basic Concepts of Step Functions

Collection of tasks◦ Worker resources – can be on premise!◦ Simple deciders

Steps◦ Pass State◦ Task State◦ Choice State◦ Wait State◦ Succeed State◦ Fail State

Multiple independent executions can run at once

JSON-based language - https://states-language.net/spec.html

11

Services Involved (Partial)CloudTrail CloudWatch Lambda

Inspector Step Functions EC2

12

Caution!Cutting, and sometimes bleeding, edge stuff here

13

Basic Flow

Approve/Reject New Image

Test the Image

Launch Test Instance

Detect New Image

14

Detect1. Track API calls - CloudTrail setup on account

2. CloudWatch Log event filter subscription tracks any image being built◦ RegisterImage (instance images)◦ CreateImage (EBS images)◦ CopyImage (copying public instances into this account)

3. CloudWatch triggers Lambda function to start Step Function◦ Input: AMI ID

4. Check Image State

Detect Launch Test Approve

15

Launch1. Launch Instance of candidate AMI

◦ Launch Instance◦ Uses an EC2 Spot Instance

2. Install Inspector Agent◦ UserData with simple “curl | bash”

3. Check Instance State

4. Tag Instance for Scanning◦ Assigns a batch ID to identify the instance(s) to scan

Detect Launch Test Approve

16

Test1. Start Inspector Run

2. Wait 15 Minutes

3. Poll Inspector Status

4. Try and Repeat with Timeout

Detect Launch Test Approve

17

Approve/Reject1. Terminate Instance

2. Parse results◦ Arbitrary criteria, are there any CVEs rated with a CVSS of 6 or above?

◦ If so, FAIL◦ Else PASS

3. Tag AMI

Detect Launch Test Approve

18

What We’ve Built So Far

19

DeploymentBASIC STEP FUNCTION WORKS…NOW WHAT?

20

Enter Terraform

21

Terraform – AMI-Security-Validator

Creates the IAM roles for Lambda functions◦ Launch EC2 Instance◦ Check Instance Ready◦ Tag EC2 Resources◦ Start Inspector Assessment Run◦ Check Inspector Assessment Run Complete◦ Parse Inspector Assessment Run Findings◦ Terminate Instances◦ Start Step Functions

Create Lambda Functions from S3 Artifacts

CloudTrail event trigger

22

Continuous Deployment with Travis

GitHub Commit Travis CI Build and Push to S3

23

Using Terraform Set up infrastructure VPC

VPC flow logs (use the VPC module)

Subnets in all three AZs

Private subnets

Internet gateways

Security group which allows access from home IP

24

Future Directions•Present a user interface that provides only tagged AMIs for launch by users

•Service Catalog?

25

Lessons Learned

26

Gripes and Lessons Learned – Step Functions

Step Functions cannot be edited once created

Eventual consistency is a very annoying problem

“Programing” in JSON

Triggering state machines…how?

Result processing is limited

Execution ID is not available to the state machine

Clearly defining the APIs of your functions is sanity saving

statelint tool is ruby based

AWS Config not supported

27

More Gripes – Lambda and Inspector

Lambda◦ Python environment only recently received updates to work

with Step Functions – No release notes!◦ AWS Config not supported◦ Python 3.5 & Tagging

Inspector◦ API is clunky – best suited for static target list◦ ResourceGroup entity required by TargetGroup, but no ability

to describe, modify, or delete them after creation…are they gone when the TargetGroup is deleted? Don't know.

◦ Ubuntu 16.04 LTS support launched only recently (Jan 5) available

◦ Preview-agents API takes a preview-agents-arn, but that's really an assessment-target-arn

◦ Strange convergence issues between resource groups, targets, and runs

28

Would You Like Some More Gripes?

Management◦ No Terraform or CloudFormation support for

Step Functions◦ Terraform support coming soon!

◦ No CloudFormation support for AWS Inspector◦ Terraform now supports AWS Inspector as of

v0.8.5

IAM Roles◦ Pathing strategy for IAM roles is really helpful◦ Placing roles under a /service-roles path makes

filtering and sorting a dream

29

Thanks! David F. Severski

david@severski.net@DSeverski

30

ReferencesLambda Code

https://github.com/davidski/infrastructure-lambdas

Image Scanner Step Function Definition & Terraform Setuphttps://gist.github.com/davidski/788afdb2a79d77edfb5fea65b50369e7

Terraform Support for Step Functionshttps://github.com/hashicorp/terraform/pull/11420

AWS Step Language Documentationhttps://states-language.net/spec.html

Packerhttps://packer.io

Terraformhttps://terraform.io