reactive reference architecture -...
TRANSCRIPT
![Page 1: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/1.jpg)
Reactive Reference Architecture
Markus Jura @markusjura
Nilanjan Raychaudhuri @nraychaudhuri
s
![Page 2: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/2.jpg)
Big dataMassive Learning Platform
http://allthingsd.com/files/2013/07/minions-‐640x327.jpg
![Page 3: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/3.jpg)
3
Principles of Reactive Systems
![Page 4: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/4.jpg)
Architecture
4
Web (Play)
backend (Akka)
Journal (Cassandra)
Fast data processing (Spark)
![Page 5: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/5.jpg)
http://cdn1.ssninsider.com/wp-‐content/uploads/2013/08/despicable-‐me-‐2-‐minions-‐e1375818315418.png
![Page 6: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/6.jpg)
Problem: Splitting application
![Page 7: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/7.jpg)
Monolithic
7
![Page 8: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/8.jpg)
Splitting MLP
8
Frontend nodes
Backend nodes
![Page 9: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/9.jpg)
Akka Distributed Pub/Sub• Location transparency
• Publish/Subscribe
• Dynamically add/remove
9
![Page 10: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/10.jpg)
Distributed Pub/Sub Mediator
10
Frontend nodes
Backend nodes
M M
M M M
![Page 11: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/11.jpg)
Code
![Page 12: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/12.jpg)
Frontend
12
// Starting up the mediatorval mediator = DistributedPubSubExtension(system).mediator
// Sending message through mediatormediator ? DistributedPubSubMediator.Send(“/user/course-aggregate”, CreateCourse(course), localAffinity = true)
![Page 13: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/13.jpg)
Backend
13
// Starting up the mediatorval mediator = DistributedPubSubExtension(system).mediator
// Setting up the servicesval courseAggregate = system.actorOf(CourseRepository.props, "course-aggregate")mediator ! DistributedPubSubMediator.Put(courseAggregate)val levelAggregate = system.actorOf(LevelRepository.props, "level-aggregate")mediator ! DistributedPubSubMediator.Put(levelAggregate)
![Page 14: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/14.jpg)
http://cdn1.ssninsider.com/wp-‐content/uploads/2013/08/despicable-‐me-‐2-‐minions-‐e1375818315418.png
![Page 15: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/15.jpg)
Problem: Distributed State
![Page 16: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/16.jpg)
State on single node
16
Frontend nodes
Backend nodes
![Page 17: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/17.jpg)
State across nodes
17
Frontend nodes
Backend nodes
![Page 18: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/18.jpg)
Akka Cluster Sharding
• Partition actors across Cluster
• Location transparency
• Rebalancing
18
![Page 19: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/19.jpg)
How does it work?
19
Shard Coordinator
Shard Region 1 Shard Region 2
![Page 20: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/20.jpg)
Code
![Page 21: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/21.jpg)
Backend: Create ShardRegion
21
// Backend: Start cluster sharing in non-proxy mode ClusterSharding(system).start( "users", Some(User.props), UserSharding.idExtractor, UserSharding.shardResolver())
![Page 22: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/22.jpg)
Backend: Shard Resolver & Id Extractor
22
// Extract the user shard// Rule of thumb: 10 times the max numbers of sharding nodesdef shardResolver(shardCount: Int = 20) = { case UserService.Envelope(email, _) => email.hashCode % shardCount.toString }
// Extract the id and payload of the user actorval idExtractor: ShardRegion.IdExtractor = { case UserService.Envelope(email, payload) => (email, payload)}
![Page 23: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/23.jpg)
Frontend
23
// Send message via shard regionshardRegion ? UserService.Envelope(email, GetUser(email))
// Start sharding in proxy mode on every frontend node val shardRegion: ActorRef = ClusterSharding(system).start( "users", None, UserSharding.idExtractor, UserSharding.shardResolver())
![Page 24: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/24.jpg)
Problem: Loosing State
![Page 25: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/25.jpg)
State in memory
25
Frontend nodes
Backend nodes
![Page 26: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/26.jpg)
Persist state
26
Frontend
Backend
![Page 27: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/27.jpg)
Akka Persistence
• Stores state in Journal DB
• Event Sourcing
• Restore state
27
![Page 28: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/28.jpg)
Code
![Page 29: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/29.jpg)
Write user state
29
class User extends PersistentActor {
// User id / sharding extractor id override def persistenceId: String = self.path.name
// Persist UserCreated event to Cassandra journal override def receiveCommand: Receive = { case UserService.CreateUser(userProgress) => persist(UserCreated(userProgress)) { userCreated => // Update user state onUserCreated(userCreated) sender() ! UserService.UserCreated } } ...
![Page 30: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/30.jpg)
Recover user state
30
...
// Replay UserCreated message to recover stateoverride def receiveRecover: Receive = { case e: UserCreated => onUserCreated(e)}
![Page 31: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/31.jpg)
http://cdn1.ssninsider.com/wp-‐content/uploads/2013/08/despicable-‐me-‐2-‐minions-‐e1375818315418.png
![Page 32: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/32.jpg)
Problem: Replicate Cache
![Page 33: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/33.jpg)
Singleton Cache
33
Frontend nodes
Backend nodes
C
![Page 34: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/34.jpg)
What is CRDT?
34
Conflict-free replicated data type is a type of specially designed data structure used to achieve strong eventual consistency
Great talk on CRDT by Sean Cribbs: Eventually Consistent Data Structures
![Page 35: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/35.jpg)
CvRDT
35
![Page 36: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/36.jpg)
CvRDT - Example
36
![Page 37: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/37.jpg)
CmRDT
37
![Page 38: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/38.jpg)
Akka Data replication• CRDT for Akka cluster
• Replicated in-memory data store
• Supports many types: (GSet, ORSet, PNCounter…)
38
![Page 39: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/39.jpg)
CRDT Replicator
39
Frontend nodes
Backend nodes
C CR R
R R R
![Page 40: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/40.jpg)
Code
![Page 41: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/41.jpg)
Frontend
41
//starting up the replicatorval replicator = DataReplication(system).replicator
//get cache entryval f = replicator ? Get(key, ReadLocal, None)
![Page 42: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/42.jpg)
Backend
42
//starting up the replicatorval replicator = DataReplication(system).replicator
//updating the cache replicator ! Update(Course.cacheKey, GSet.empty, WriteLocal)
(_ + courseCreated.course)
![Page 43: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/43.jpg)
Q & Option[A]
@nraychaudhuri@markusjura
![Page 44: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/44.jpg)
44
React.js
frontend
REST API Actors
backend
Cache
PubSub CRDTPersistenceCluster Sharding
journal
![Page 45: reactive reference architecture - Lightbenddownloads.typesafe.com/website/presentations/ScalaDaysSF... · 2015-03-25 · Backend: Shard Resolver & Id Extractor 22 // Extract the user](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c5b727e708231d435002a/html5/thumbnails/45.jpg)
©Typesafe 2014 – All Rights Reserved