serverless architectural patterns · amazon kinesis streams and aws lambda •number of amazon...

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

G O T OAmsterdam 2018

Serverless Architectural PatternsBoaz Ziniman, Technical Evangelist – Amazon Web Service

@ziniman

boaz.ziniman.aws


GOTOAmsterdam

Serverless Beyond Lambda


GOTOAmsterdam

Pattern 1: Web App/Microservice/API

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Web application

Data stored in Amazon

DynamoDB

Dynamic content in AWS Lambda

Amazon API Gateway

Browser

Amazon CloudFront

Amazon S3

Amazon Cognito


Amazon API Gateway AWS

LambdaAmazon

DynamoDB

AmazonS3

Amazon CloudFront

• Bucket Policies• ACLs

• Origin Access Identity (OAI)• Geo-Restriction• Signed Cookies• Signed URLs• DDOS Protection

IAM IAM

Serverless web app security

• Throttling• Caching• Usage Plans• ACM

Static Content

Browser

Amazon Cognito


Custom Authorizer LambdafunctionClient

Lambdafunction

Amazon APIGateway

AmazonDynamoDB

AWS Identity &Access Management

SAMLTwo types:• TOKEN - authorization token

passed in a header• REQUEST – all headers, query

strings, paths, stage variables or context variables.Custom

AuthorizerCognitoUser Pool

Custom Authorizers


Multi-Region with API Gateway

us-west-2

us-east-1

Client

AmazonRoute 53

RegionalAPI

Endpoint

RegionalAPI

Endpoint

Custom Domain Name

Custom Domain Name

API Gateway

API Gateway

Lambda

Lambda

api.m

ycor

p.co

m

CNAME

CNAME


GOTOAmsterdam

http://bit.ly/ServerlessShophttps://github.com/patrick-michelberger/serverless-shop

http://bit.ly/ServerlessShop

https://github.com/patrick-michelberger/serverless-shop


GOTOAmsterdam

Pattern 2: Stream Processing


Stream processing characteristics

• High ingest rate• Near real-time processing (low latency from ingest to

process)• Spiky traffic (lots of devices with intermittent network

connections)• Message durability• Message ordering


Streaming data ingestion

Amazon CloudWatch:Delivery metrics

Amazon S3:Buffered files

Kinesis Agent

Record Producers Amazon Redshift:

Table loads

Amazon Elasticsearch Service:Domain loads

Amazon S3:Source record backup

AWS Lambda:Transformations &

enrichment

Amazon DynamoDB:Lookup tables

Raw records

Lookup

Transformed records

Transformed recordsRaw records

Amazon Kinesis Firehose:Delivery stream


Best practices

• Tune Firehose buffer size and buffer interval• Larger objects = fewer Lambda invocations, fewer S3 PUTs

• Enable compression to reduce storage costs

• Enable Source Record Backup for transformations• Recover from transformation errors

• Follow Amazon Redshift Best Practices for Loading Data• How to handle time series, sorted data

http://docs.aws.amazon.com/redshift/latest/dg/c_loading-data-best-practices.html


bit.ly/AWSIoTColours


Colour cube voting

Amazon S3 Bucket

Amazon CloudFront

AWS IoT

AmazonCognito

AmazonCognito


Colour cube voting

Amazon S3 Bucket

Amazon CloudFront

AWS IoT

AmazonCognito

AmazonCognito

Amazon Elasticsearch

Amazon Kinesis

Firehose


Real-time analytics

Amazon Kinesis Streams:Ingest stream

Amazon Kinesis Analytics:Time window aggregation

Amazon Kinesis Streams:Aggregates stream

Amazon Kinesis Firehose:Error stream

Amazon S3:Error records

Record Producers

AWS Lambda:Alert function

Amazon DynamoDB:Device thresholds

AWS SNS:Notifications


Amazon Kinesis Streams and AWS Lambda

• Number of Amazon Kinesis Streams shards corresponds to concurrent invocations of Lambda function

• Batch size sets maximum number of records per Lambda function invocation

Amazon Kinesis:Stream

AWS Lambda: Processor function

Streaming source Other AWS services


Fan-out pattern

Fan-out pattern trades strict message ordering vs higher throughput & lower latency

Amazon Kinesis:Stream

Lambda: Dispatcher function

Lambda: Processor function

Increase throughput, reduce processing latency

Streaming source


Thomson Reuters – Product Insight

Solution for usage analysis tracking: Capture, analyze, and visualize analytics data generated by offerings, providing insights to help product teams continuously improve the user experience

Throughput: Tested 4,000 requests / secondGrowing to 10,000 requests / second or 25 Billion requests / month

Latency: new events to user dashboards in less than 10 seconds

Durable: no data loss since inception


GOTOAmsterdam

Pattern 3: Operations Automation


Automation characteristics

• Periodic jobs• Event triggered workflows• Enforce security policies• Audit and notification• Respond to alarms• Extend AWS functionality

… All while being Highly Available, Scalable and Auditable


AWS Ops Automator

Amazon CloudWatch:Time-based events

AWS Lambda:Event handler

AWS Lambda:Task executors

AWS SNS:Error and warning notifications

Resources in multiple AWS Regions and Accounts

Amazon EC2 Instances

Tags

OpsAutomatorTaskList CreateSnapshotAmazon DynamoDB:Task configuration & tracking

Amazon CloudWatch:Logs

Amazon Redshift Clusters

https://aws.amazon.com/answers/infrastructure-management/ops-automator/

https://aws.amazon.com/answers/infrastructure-management/ops-automator/


Image recognition and processing

Web App

Amazon DynamoDB:Image meta-data & tags

Amazon Cognito:User authentication

Amazon S3:Image uploads

AWS Step Functions: Workflow orchestration

Start state machine execution

1

Extract image meta-data2

Amazon Rekognition: Object detection

Invoke Amazon Rekognition

Generate image thumbnail

3

3Store meta-data and tags

4

https://github.com/awslabs/lambda-refarch-imagerecognition

https://github.com/awslabs/lambda-refarch-imagerecognition


Step Functions state machine


Enforce security policies

RDP from 0.0.0.0/0

RDP from 0.0.0.0/0

CloudWatch Event Bus in another AWS Account

New Security Group ingress rule Amazon CloudWatch Events:Rule

AWS Lambda:Remediate and alert

AWS SNS:Email alert

Ingress rule deleted


Autodesk - Tailor

Serverless AWS Account Provisioning and Management Service:• Automates AWS Account creation,• Configures IAM, CloudTrail, AWS Config, Direct Connect, and VPC• Enforces corporate standards• Audit for compliance

Provisions new Accounts in 10 minutes vs 10 hours in earlier manual process

Open source and extensible: https://github.com/alanwill/aws-tailor

https://github.com/alanwill/aws-tailor


Best practices

• Gracefully handle API throttling by retrying with an exponential back-off algorithm (AWS SDKs do this for you)

• Publish custom metrics from your Lambda function that are meaningful for operations (e.g. number of EBS volumes snapshotted)

• Enable X-Ray tracing for your Lambda functions

• Document how to disable event triggers for your automation when troubleshooting


GOTOAmsterdam

Additional Patterns


Serverless Data Lake

S3 Bucket(s)

Key Management

Service

Amazon Athena

AWSCloudTrail

AmazonCognito

AWS IAM

Amazon Kinesis

Streams

Amazon Kinesis

Firehose

Amazon ES

Amazon QuickSight

AWS GlueAmazon

DynamoDB

Amazon Macie

Amazon API Gateway

AWS IAM

Amazon Redshift

Spectrum

AWS Direct

Connect

Ingest

Catalog & Search

Security & Auditing

API/UI

Analytics & Processing

AWS GlueAWSLambda


Search and Data Catalog

• DynamoDB asMetadata repository

• Amazon Elasticsearch

Catalog & Search

AWS Lambda

AWS Lambda

Metadata Index(DynamoDB)

Search Index(Amazon ES)

ObjectCreatedObjectDeleted PutItem

Update Stream

Update Index

Extract Search Fields

S3 Bucket

https://aws.amazon.com/answers/big-data/data-lake-solution/

https://aws.amazon.com/answers/big-data/data-lake-solution/


Further Reading

• Optimizing Enterprise Economics with Serverless Architectureshttps://d0.awsstatic.com/whitepapers/optimizing-enterprise-economics-serverless-architectures.pdf

• Serverless Architectures with AWS Lambdahttps://d1.awsstatic.com/whitepapers/serverless-architectures-with-aws-lambda.pdf

• Serverless Applications Lens - AWS Well-Architected Frameworkhttps://d1.awsstatic.com/whitepapers/architecture/AWS-Serverless-Applications-Lens.pdf

• Streaming Data Solutions on AWS with Amazon Kinesishttps://d1.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf

• AWS Serverless Multi-Tier Architectureshttps://d1.awsstatic.com/whitepapers/AWS_Serverless_Multi-Tier_Archiectures.pdf

https://d0.awsstatic.com/whitepapers/optimizing-enterprise-economics-serverless-architectures.pdf

https://d1.awsstatic.com/whitepapers/serverless-architectures-with-aws-lambda.pdf

https://d1.awsstatic.com/whitepapers/architecture/AWS-Serverless-Applications-Lens.pdf

https://d1.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf

https://d1.awsstatic.com/whitepapers/AWS_Serverless_Multi-Tier_Archiectures.pdf


Summary

Use DevOps tools to automate your serverless deployments

Apply serverless patterns for common use-cases:• Web application• Data Lake Foundation• Stream processing• Operations automation

What will you build with Serverless?


Boaz Ziniman – Technical Evangelist, [email protected]

serverless architectural patterns · amazon kinesis streams and aws lambda •number of amazon...

Documents