cliff moon - building polyglot distributed systems with scalang, boundary tech talks october 6, 2011
DESCRIPTION
Erlang and Scala have a very complementary set of strengths and weaknesses for building large scale distributed systems. Cliff shows how to build polyglot distributed systems using Scalang, a library that allows actors written in Scala to perform remote messaging with both Erlang and other Scala nodes. Presented at Boundary's Pizza, Beer, & Tech Talks meetup on October 6, 2011.TRANSCRIPT
![Page 1: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/1.jpg)
Building Polyglot Systems With ScalangCliff Moon@moonpolysoft
Thursday, September 22, 2011
![Page 2: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/2.jpg)
Why Scala and Erlang?
• Complementary sets of features.
• Compatible type systems.
• Separates concerns.
Thursday, September 22, 2011
![Page 3: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/3.jpg)
metermetermetermetermetermetermetermetermetermetermetermeter
Collector
Collector
streaker
scylla
REST RESTRESTREST
Boundary ArchitectureThursday, September 22, 2011
![Page 4: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/4.jpg)
First Revision
• JInterface and wrappers.
• Cumbersome code.
• Wrong level of abstraction.
• Not performant.
Thursday, September 22, 2011
![Page 5: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/5.jpg)
JInterface Wrappers
def task(m : OtpErlangObject, mbox : OtpMbox) = m match { case ETuple(EAtom("cluster"), pid : Pid, ref : Ref) =>
Thursday, September 22, 2011
![Page 6: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/6.jpg)
Scalang
• Correctness of behavior.
• Performance.
• Simplicity.
Thursday, September 22, 2011
![Page 7: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/7.jpg)
Netty
Jetlang
NIO Sockets
Proc Proc Proc
Codecs
Delivery
Scalang ArchitectureThursday, September 22, 2011
![Page 8: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/8.jpg)
Using Scalang
Thursday, September 22, 2011
![Page 9: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/9.jpg)
Node
//make sure epmd is runningval node = Node(“[email protected]”, “cookie”)
Thursday, September 22, 2011
![Page 10: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/10.jpg)
Processes
class Echo(ctx : ProcessContext) extends Process(ctx) {override def onMessage(msg : Any) = msg match { case (pid : Pid, m : String) => pid ! m}
}val echoPid = node.spawn[Echo](“echo_server”)
Thursday, September 22, 2011
![Page 11: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/11.jpg)
Sending Messages
• Send messages to a Pid.
• Send messages to a local registered name.
• Send messages to a remote registered name.
Thursday, September 22, 2011
![Page 12: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/12.jpg)
Sending Messages - in process
aPid ! ‘do_something
‘regname ! ‘do_another_thing
(‘regname, ‘[email protected]) ! ‘do_something_else
Thursday, September 22, 2011
![Page 13: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/13.jpg)
Sending Messages - outside proc
node.send(aPid, ‘do_something)
node.send(‘regname, ‘do_another_thing)
node.send( (‘regname, ‘[email protected]), ‘do_something_else)
Thursday, September 22, 2011
![Page 14: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/14.jpg)
Error Handling
• Uncaught exceptions in process code.
• Implemented via bi-directional links.
• Link breakage triggers exit notification.
• Links work both intra and inter - node.
• Not pre-emptive on the Scalang side.
Thursday, September 22, 2011
![Page 15: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/15.jpg)
Error Handling
class ExitHandler(ctx : ProcessContext) extends Process(ctx) { override def trapExit(from : Pid, msg : Any) { log.warn(“exit notification from %s”, from) }}
Thursday, September 22, 2011
![Page 16: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/16.jpg)
Type MappingsFrom Erlang To Scala From Scala To Erlang
Small Integer Int Byte Small Integer
Integer Int Int Integer
Float Double Long Small Bignum
Boolean Boolean Double Float
Atom Symbol Symbol Atom
Reference Reference Reference Reference
Port Port Port Port
Pid Pid Pid Pid
Small Tuple Tuple Fun Fun
Large Tuple BigTuple String String
String String List List
List List BigInteger Large Bignum
Binary ByteBuffer Array[Byte] Binary
Small Bignum Long ByteBuffer Binary
Large Bignum BigInt BitString Bitstring
Fun Fun Tuple Tuple
Bitstring Bitstring BigTuple Tuple
Thursday, September 22, 2011
![Page 17: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/17.jpg)
Rich Type Mappings
• Invoked for tagged tuples.
• User-supplied decode logic.
• Avoids performance penalty of reflective instantiation.
Thursday, September 22, 2011
![Page 18: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/18.jpg)
Rich Type Mappings
(‘struct, List( (‘key, value), (‘key2, value2), ... ))
{struct, [ {key, Value}, {key2, Value2}, ... ]}
Thursday, September 22, 2011
![Page 19: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/19.jpg)
Rich Type Mappings
object StructFactory extends TypeFactory { def createType(name : Symbol, arity : Int, reader : TermReader) : Any = { reader.mark (name, arity) match { case (‘struct,2) => Some(readMap(reader)) case _ => reader.reset; None } }}
Thursday, September 22, 2011
![Page 20: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/20.jpg)
Rich Type Mappings
val node = Node(name,cookie,NodeConfig( typeFactory = StructFactory))
Thursday, September 22, 2011
![Page 21: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/21.jpg)
Services
• Initialization parameters.
• Built in call and cast support.
• Transparent interoperability with gen_server.
Thursday, September 22, 2011
![Page 22: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/22.jpg)
Services
• handleCall - gen_server:call/2 - request & respose
• handleCast - gen_server:cast/2 - request
• handleInfo - Raw message received.
Thursday, September 22, 2011
![Page 23: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/23.jpg)
Services
case class EchoArgs(name : Symbol)class Echo(ctx : ServiceContext[EchoArgs]) extends Service(ctx) { val EchoArgs(name) = ctx.args override def handleCall(from : (Pid,Reference), req : Any) : Any = { (name, req) }}
Thursday, September 22, 2011
![Page 24: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/24.jpg)
Anonymous Processes
node.spawn { mbox => val msg = mbox.receive // do some long running work node.send(‘master, (results, mbox.self) )}
Thursday, September 22, 2011
![Page 25: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/25.jpg)
Runtime Metrics
• Message meters per connection.
• Execution histograms per process.
• Serialization time histograms.
• Message queue size gauges.
• Internal thread pool metrics.
Thursday, September 22, 2011
![Page 26: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/26.jpg)
Runtime MetricsThursday, September 22, 2011
![Page 27: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/27.jpg)
Performance Tuning
• ThreadPoolFactory - pluggable thread pool implementations.
• Elastic thread pools from overlock.
• Threads tuned to max(8, cpu_count * 2).
• Beware of starvation.
Thursday, September 22, 2011
![Page 28: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/28.jpg)
Thread Pools
• Boss pool - Initial connection and accept handling.
• Worker pool - Non blocking reads and writes.
• Actor pool - Process callbacks.
• Batch Executor - Per-process execution logic.
Thursday, September 22, 2011
![Page 29: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/29.jpg)
Thread Pools
val node = Node(name,cookie,NodeConfig( poolFactory = MyThreadPools))
Thursday, September 22, 2011
![Page 30: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/30.jpg)
Process Patterns
Thursday, September 22, 2011
![Page 31: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/31.jpg)
Deferred Reply
def handleCall(from : (Pid, Reference), msg : Any) : Any = { ctx.node.spawn { mbox => // long running work reply(from, response) } ‘noreply}
Thursday, September 22, 2011
![Page 32: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/32.jpg)
State Machine
class Stateful(ctx : ProcessContext) extends Process(ctx) { @volatile var state = ‘a override def onMessage(msg : Any) = (msg, state) match { case (‘event, ‘a) => ... }}
Thursday, September 22, 2011
![Page 33: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/33.jpg)
Worker Pools
class StatelessWorker(ctx : ProcessContext) extends Process(ctx) { def onMessage(msg : Any) = msg match { ... }}
//experimental!node.spawn[StatelessWorker](reentrant = true)
Thursday, September 22, 2011
![Page 34: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/34.jpg)
Supervision Trees
class TaskMaster(ctx : ProcessContext) extends Process(ctx) { def onMessage(msg : Any) = { //distribute work to workers }
override def handleExit(worker : Pid, reason : Any) { //remove worker from pool, or restart }}
Thursday, September 22, 2011
![Page 35: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/35.jpg)
Admin Endpoint
class Admin(ctx : ServiceContext[Args]) extends Service(ctx) { def handleCall(from: (Pid,Reference), req : Any) = req match { case (‘reassign, node : Symbol) => //reassign message stream to new node }}
Thursday, September 22, 2011
![Page 36: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/36.jpg)
Admin Endpoint
ok = gen_server:call({admin, backend@data01}, {reassign, cruncher@data02}).
Thursday, September 22, 2011
![Page 37: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/37.jpg)
Demo!
Thursday, September 22, 2011
![Page 38: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/38.jpg)
Conclusion
• Wrap services in Scalang actors.
• Throw out your RPC codegen tools.
• Embrace a mesh topology.
• Let Erlang and Scala do the things they are good at.
Thursday, September 22, 2011
![Page 39: Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011](https://reader035.vdocuments.us/reader035/viewer/2022070323/559c1b281a28ab18598b4841/html5/thumbnails/39.jpg)
Questions?https://github.com/boundary/scalanghttps://github.com/boundary/overlockThank you.
Thursday, September 22, 2011