![Page 1: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/1.jpg)
Putting the “Micro” Backin Microservices
Sol Boucher, Carnegie Mellon University
Joint work with:Anuj Kalia
David G. AndersenMichael Kaminsky, Intel Labs
![Page 2: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/2.jpg)
Tech Target
Wall Street Journal
The Register
Hacker Noon
InfoWorld 2
![Page 3: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/3.jpg)
The hope for serverless computing
3
Only have to manage code
Microservices invoked by triggers
Microservices are stateless
This makes the system scalable
Fine-grained billing that scales to zero
![Page 4: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/4.jpg)
Median AWS Lambda warm-start latency 25 ms
Median cold-start latency >160 ms
Median AWS Lambda warm-start latency 25 ms
Median cold-start latency >160 ms
Goal: Reduce microservice invocation latency
4
[Yesterday, ATC‘18]
Latency between Azure VMs ~10 μs [AccelNet, NSDI‘18]
Commit ACID transactions in ~20 μs [FaRM, SOSP‘15]
Latency between Azure VMs ~10 μs [AccelNet, NSDI‘18]
Commit ACID transactions in ~20 μs [FaRM, SOSP‘15]
![Page 5: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/5.jpg)
Speed begets generality
Make it fast, rather than general or powerful.
— Butler Lampson
“”
5
![Page 6: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/6.jpg)
Worker node
Current request path
6
μservice
μservice
Dispatcher process μservice
![Page 7: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/7.jpg)
Proposal: Reduce overhead...
Worker node
7
CPU core
CPU core
CPU core
CPU core
μsvc μsvc μsvc μsvc μsvc μsvc
μsvc μsvc μsvc μsvc μsvc μsvc
![Page 8: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/8.jpg)
Proposal: ...by running code in shared workers...
Worker node
8
CPU core
CPU core
CPU core
CPU core
Worker process
μservice μservice
Worker process
μservice μservice
Worker process
μservice μservice
![Page 9: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/9.jpg)
Proposal: ...and distributing work using polling
Worker node
9
CPU core
CPU core
CPU core
CPU core
Worker process
μservice μservice
Worker process
μservice μservice
Worker process
μservice μservice
Dispatcher process
![Page 10: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/10.jpg)
Proposal: ...and distributing work using polling
Worker node
10
CPU core
CPU core
CPU core
CPU core
Worker process
μservice μservice
Worker process
μservice μservice
Worker process
μservice μservice
Dispatcher process
But...How do we provide isolation?
![Page 11: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/11.jpg)
We use Rust for this, inspired by NetBricks and [OSDI‘16] [SOSP‘17]
How do we achieve isolation similar to processes?
Language-based isolation: compile-time safety guarantees
Fine-grained preemption: intra-process task interruption
11
User submits Rust code; we verify it
![Page 12: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/12.jpg)
Language-based isolation cuts invocation latency
12
![Page 13: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/13.jpg)
Language-based isolation cuts invocation latency
13
![Page 14: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/14.jpg)
Language-based isolation: Use Rust
Rust is…● Strongly typed, compiled● Specified safe subset● No garbage collector
Memory safety guarantees:● No dereferencing null/dangling pointers● All variables initialized to valid values● Enforced data immutability
14
![Page 15: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/15.jpg)
Worker node
Language-based isolation: Defense in depth
15
Worker process
μservice μservice Blacklisted library functions
User
Kernelseccomp() to permit only whitelisted system calls
![Page 16: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/16.jpg)
Worker node
Language-based isolation: Defense in depth
16
Worker process
μservice μservice Blacklisted library functions
User
Kernelseccomp() to permit only whitelisted system callsBut...
What if a microservice doesn’t yield?
![Page 17: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/17.jpg)
CPU timesharing: Fine-grained preemption
Goal: Recover from microservice that doesn’t return quickly
1. Regain control of the CPU2. Abort/clean up after microservice’s code
Implementation: POSIX timers, special cleanup logic
17
![Page 18: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/18.jpg)
Fine-grained preemption
18
3-μs period is possible!
20-μs period is practical!
Preemption interval (μs)
Workload throughput
(M ops/s)
BaselinePreemption90% of Baseline
![Page 19: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/19.jpg)
Fine-grained preemption: Aborting and cleanup
SIGALRM handler: missed deadline?
Worker’s main loop catches exception
Handler returns, microservice
continuesno
Handler throws exception,
unwinding stack
yes
19
![Page 20: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/20.jpg)
Trust model
Trusted computing base:
● Rust compiler, standard library● Any allowed unsafe or native dependencies
Successful compilation indicates microservice is memory safe
Successful linking indicates all dependencies are trusted
20
![Page 21: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/21.jpg)
✓ Consolidate microservices into shared processes✓ Improved local invocation latency by orders of magnitude✓ (Hopefully) better resource utilization
→ Current limitations and future work
21
Recap
![Page 22: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/22.jpg)
Call to malloc()
Future work: Aborting/cleanup limitations
22
Worker process
μservice μservice
![Page 23: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/23.jpg)
Call to malloc()
Future work: Aborting/cleanup limitations
23
Worker process
μservice μservice
![Page 24: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/24.jpg)
Call to malloc()
Future work: Aborting/cleanup limitations
24
Worker process
μservice μserviceμservice
![Page 25: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/25.jpg)
Call to malloc()
Future work: Aborting/cleanup limitations
25
Worker process
μservice μserviceμservice μservice
![Page 26: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/26.jpg)
Upcoming: More general accounting/deallocation scheme
● Operates outside the Rust runtime● Disables preemption during trusted library routines
Call to malloc()
Future work: Aborting/cleanup limitations
26
Worker process
μservice μserviceμservice μservice
![Page 27: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/27.jpg)
Future work: Side-channel attacks
Heightened Spectre vulnerability requires hardware mitigation
Must consider microservices’ access to:
● Process’s proximity to resource limits● Addresses and timings from the dynamic allocator● File descriptor numbers
Shorter microservice durations make behavior less obscure
27
![Page 28: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/28.jpg)
Conclusion
Improved performance by shifting isolation abstraction layer
Replaced traditional process-based isolation with:
● Language-based isolation● Fine-grained preemption
28
![Page 29: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/29.jpg)
Conclusion
Improved performance by shifting isolation abstraction layer
Replaced traditional process-based isolation with:
● Language-based isolation● Fine-grained preemption
29
Questions?
![Page 30: Intel Labs Putting the “Micro” Back David G. …...Median AWS Lambda warm-start latency 25 msMedian cold-start latency Goal: Reduce microservice invocation latency>160 ms 4 [Yesterday,](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e860e193053e413d802a0d7/html5/thumbnails/30.jpg)
Thank you!
30