![Page 1: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/1.jpg)
Will Norman
Lessons Learned Building and Operating a Serverless Data Pipeline
![Page 2: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/2.jpg)
![Page 3: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/3.jpg)
Introduction
● Will Norman - Director of Engineering @ Intent○ FinTech and AdTech background
● Intent○ Data Science company for commerce sites○ Primary application is an ad network for travel sites
![Page 4: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/4.jpg)
● MOD owns data
● 4 Engineers
● 1 Product Manager
MOD Squad
![Page 5: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/5.jpg)
What we’ll be covering
● What is Serverless?
● Intent Data Platform
● Lessons Learned
![Page 6: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/6.jpg)
● More about managed services than lack servers
● Not just FaaS
● Scale on demand / pay for only what you use
● Empowers developers to own their platform
What is Serverless?
![Page 7: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/7.jpg)
● Active MQ
● Log Processors○ Java applications
○ Kept state locally
○ Cron scheduled tasks to roll files to S3
○ Ran on dedicated EC2 instances
● S3
Intent Data Platform [Old World]
![Page 8: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/8.jpg)
![Page 9: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/9.jpg)
● Kinesis
● Lambda
● Kinesis Firehose
● SNS
● AWS Batch
● S3
Intent Data Platform [New World]
![Page 10: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/10.jpg)
![Page 11: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/11.jpg)
![Page 12: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/12.jpg)
● Streaming Data Consumers
● Spark Jobs / Aggregations -> Redshift
● Snowflake Loader -> Snowflake
● Parqour -> Athena○ EMR based jobs that convert AVRO -> Parquet
Data Consumers
![Page 13: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/13.jpg)
● Fewer production issues
● Separation of concerns
● Horizontally scalable
● Removed a lot of undifferentiated heavy lifting
Worth the move?
![Page 14: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/14.jpg)
Lessons Learned
1. Total Cost of Ownership
2. Think about data formats upfront
3. Design for Failure
4. Design for Scalability
5. Not NoOps just DiffOps
6. Build Components
7. CI / CD Strategies
8. Leverage the Community
![Page 15: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/15.jpg)
● On demand costs
● Hidden Costs / Tag All The Things!
● Enterprise Support
● Value of being able to focus on core business problems
Total Cost of Ownership
![Page 16: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/16.jpg)
● What does the ecosystem support?
● Schema vs Schemaless (eg AVRO vs JSON)
● Data validation & Data evolution
● Data at rest vs data in flight
● JSON / CSV / AVRO / Parquet?
Think about data formats up front
![Page 17: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/17.jpg)
record DataWrapper { string dataType; long schemaFingerprint; bytes data; }
● Publish Schema in JSON format to S3● Consumers lookup schemas, and calculate fingerprints
Schema Registry
![Page 18: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/18.jpg)
● System Guarantees?
● Idempotency
● Over process (data lookbacks)
● Dead Letter Queues
Design for failure
![Page 19: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/19.jpg)
![Page 20: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/20.jpg)
![Page 21: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/21.jpg)
● Decouple from non-scalable systems
● Don’t run lambdas in VPC if you can help it
● Partition data at rest
● Shard events based on GUID / random id if ordering isn’t
necessary
● Think about fan out patterns
Design for Scalability
![Page 22: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/22.jpg)
![Page 23: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/23.jpg)
● Application problem or service problem
● Platform Limits
● Logs
● Metrics
● Dashboards
● Alerts
Not NoOps, just DiffOps
![Page 24: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/24.jpg)
Some things remain the same
![Page 25: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/25.jpg)
![Page 26: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/26.jpg)
● Help to reason about different parts of the system
● Make it easy to do the right thing
● Easier to extend
● Infrastructure as Code
Build Components
![Page 27: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/27.jpg)
module "conversion_event_processor" {
source = "../modules/event_processor"
data_type = "conversion"
data_source = "ad_server"
processor_lambda_handler = "com.intentmedia.data.stream.ConversionLambda::handler"
environment = "${var.environment}"
firehose_lambda_handler = "com.intentmedia.data.stream.ConversionFirehose::handler"
processor_lambda_reserved_concurrent_executions = 3
firehose_lambda_reserved_concurrent_executions = 2
}
![Page 28: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/28.jpg)
● Step backwards from being able to run stack locally
● Unit tests for business logic
● Integration Tests / End to End tests to ensure that
everything is working as expected
● Use different AWS accounts to segregate staging and
production
CI / CD
![Page 29: Serverless Data Pipeline Lessons Learned Building and ...€¦ · Lessons Learned Building and Operating a Serverless Data Pipeline. Introduction ... Serverless Forum](https://reader033.vdocuments.us/reader033/viewer/2022042218/5ec45c68e8f156666a719cbb/html5/thumbnails/29.jpg)
● Slack○ Serverless Forum○ og-aws
● Blogs○ Symphonia https://www.symphonia.io/○ Charity Majors https://charity.wtf/○ Jeremy Daly https://www.jeremydaly.com/
● Twitter● Meetup Events / Conferences
Leverage the Community