real world scala hakking nljug jfall 2011
TRANSCRIPT
![Page 1: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/1.jpg)
Real World Scala hAkking
Raymond Roestenburg
![Page 2: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/2.jpg)
About me
• Java since 1997• Scala since 2010• Mostly backend systems• Lead architect Traffic Management at CSC• Akka committer
code: http://github.com/RayRoestenburg
blog: http://roestenburg.agilesquad.com
twtr: @RayRoestenburg
![Page 3: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/3.jpg)
Company I work for
Traffic Management• Section Control• Weigh in Motion• Traffic Information• Traffic Enforcement
![Page 4: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/4.jpg)
Agenda
• @MIGO-BORAS sensordomain• Sensor networks• Typesafe stack• Akka Actors• Apache Camel Integration• Remote Actors• Monitoring• Testing• Deployment
![Page 5: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/5.jpg)
@MIGO-BORAS
• Koninklijke Marechaussee• MTV (Mobiel Toezicht Vreemdelingen)• Prevent illegal border crossing• Sensor domain• Fixed locations• Mobile vehicles• Cameras, Radars• Vehicle properties• January 2012
![Page 6: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/6.jpg)
Sensor networks
• Usable for many applications in our field• Distributed along highways and provincial roads• Traffic Enforcement• Traffic Information• Other applications• Roadside systems• Range of sensor types
– Camera’s, Infrared, Laser, Piezoelectric, Loops, Mobile units, …
• Remotely Configure & Control
![Page 7: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/7.jpg)
Why Scala & Akka
• Less code, less bugs• Less code, same VM power• Experienced team (Java, C#, C)
• 3 experienced Java devs • 1 experienced C# dev with no Java experience
• Need to interoperate with existing Java and C (JNA)
• Simplify development (C->Java->Scala)• Ubuntu
![Page 8: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/8.jpg)
Scala 2.9.x
• Learning curve (Start with simple features)• Traits, Tupels, Stackable Traits• First Class Functions• Scala Collections
• map, flatMap, partition, foreach, filter, groupBy..
• Option type• map.. orElse, foreach,..
• Case classes • REPL, DSLs, Extension methods
![Page 9: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/9.jpg)
Typesafe Stack
• Current Stack• Open Source• SBT 0.7.7• Scala 2.9.0-1• Akka 1.1.3• Camel & Camel Components 2.7.0
(FTP,Mina,Jetty,…)• Jboss Netty 3.2.4
• When we started early 2010• Akka 0.7, Scala 2.8.0Beta1
![Page 10: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/10.jpg)
Akka Usage
Core
Actors, Remote, TestKit, Serialization, STM, Agents, Dataflow, Async HTTP (Mist), FSM, Transactors
Modules
MicroKernel, Camel, AMQP, Scalaz, Spring, OSGi, DataFlow, Persistence,
![Page 11: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/11.jpg)
Akka Actors
Dispatcher
ActorRef Mailbox Actor
Threads
val a = actorOf(new Act())
a ! msgMessage to send
![Page 12: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/12.jpg)
Akka Actors
Dispatcher
Mailbox
Threads
ActorActorRef
class Act extends Actor { def receive = { case msg:Msg => { // we are here } }}
![Page 13: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/13.jpg)
Akka Actors
Dispatcher
Mailbox
Threads
ActorActorRef
![Page 14: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/14.jpg)
Akka Actors
Dispatcher
Mailbox
Threads
ActorActorRef
![Page 15: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/15.jpg)
Akka Actors
Dispatcher
Mailbox
Threads
ActorActorRef
![Page 16: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/16.jpg)
Akka Actors
Dispatcher
Mailbox
Threads
ActorActorRef
![Page 17: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/17.jpg)
One-way messaging
class Act( next : ActorRef ) extends Actor { def receive = { case msg : Msg => { val newMsg = doSomeStuff(msg) next ! newMsg } }}
SharedDispatcher
![Page 18: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/18.jpg)
Two-way
case msg : Msg => { val q = doSomeStuff(msg) next !! q match { case Some(reply) => { //receive reply } case None => .. } }
Q R
case q : Request => { val r = doSomeStuff(q) self.reply_?(q) }
![Page 19: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/19.jpg)
Two-way with Futures
val future = actorRef !!! msg failure { case e: SomeException=> failureResponse}// doesn’t blockfuture.onComplete { f=> // do your response handling… // response can be failureResponse val response = f.get}.onTimeout { => //** Akka 1.2 // do your timeout handling here}
![Page 20: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/20.jpg)
Camel Actors
• Camel Component integration• Consumers and Producers
class Consume(uri:String) extends Actor with Consumer { def endpointUri = uri def receive = { case msg: Message => { // receiving a Camel Message, extract body val data = msg.bodyAs[String] … self.reply("OK") } }}
![Page 21: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/21.jpg)
Camel Consumers
class Consume(uri:String) extends Actor with Consumer { def endpointUri = uri def receive = { case msg: Message => { // receiving a Camel Message, extract body val data = msg.bodyAs[String] … self.reply("OK") } }}
![Page 22: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/22.jpg)
Camel Producers
class Produce(uri:String) extends Actor with Producer { def endpointUri = uri def receiveBeforeProduce = { // you can change the message here before sending case msg => msg }}
![Page 23: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/23.jpg)
Camel Producers
• Use CamelContext
val context = CamelContextManager.context
val ex = context.get.getEndpoint(endpointUri).createExchangeval in = ex.getInin.setHeaders(JavaConversions.mapAsJavaMap(headers))in.setBody(body)ex.setIn(in)CamelContextManager.template.get.asyncCallback(endpointUri, ex, logOnCompetion)
![Page 24: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/24.jpg)
Dispatchers
class Endpoint(dispatcher: MessageDispatcher = Dispatchers. globalExecutorBasedEventDrivenDispatcher) extends Actor { self.dispatcher = dispatcher ….}
ThreadBasedDispatcher
ExecutorBasedEventDrivenDispatcher
ExecutorBasedWorkStealingDispatcher
PriorityExecutorBasedEventDrivenDispatcher
RollYourOwn
![Page 25: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/25.jpg)
Dispatchers
Local Processing Camel ConsumersCamel Producers
Heartbeat,Cleanup,Schedule,Long Running,Redelivery,Remote Error Handling
Remote Actors
![Page 26: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/26.jpg)
Messages
• Case classes• Simple data containers• toProtobuf and fromProtobuf on case class
and companion object for remoting
• Every msg has a • correlation Id to the event• Timestamp
• Logging, Tracing, Performance
![Page 27: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/27.jpg)
Event Process Chains
Load Balancer
Forwarding Actor
EndPoint (Consumer/Producer)
![Page 28: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/28.jpg)
Remote Actors
• Client• Actor.remote.actorFor(name, host,port)
• Server• Actor.remote.register(name, actorRef)
• remoteActorFor: (String, String, Int) => ActorRef = remote.actorFor
![Page 29: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/29.jpg)
Remote Actors Internals
• RemoteActorRef• RemoteClient• Netty Channels• Serialization (Scala,
Java, (S)JSON, ProtoBuf
• RemoteServer• Netty Server• Lifecycle Event
listeners• RemoteProtocol
ActorRef Mailbox ActorRemoteActorRef
RemoteClient RemoteServer
RemoteProtocol
![Page 30: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/30.jpg)
Remote Actors (Akka 1.1.3)
• JBoss Netty (Channels)• Not as transparent as you would like• Remote Lifecycle Listeners• Closing sockets, reconnect• Own Guaranteed Delivery implementation
– Specific requirements– Bases on Idempotent Receiver and Repeating
messages after reconnect– Non-Trivial– Heartbeats– Exp Backoff
![Page 31: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/31.jpg)
Remote configuration
• #compression-scheme = "zlib“ #leave out• zlib-compression-level = 0• client: reconnection-time-window = 2000000• client: read-timeout (we use 60)• backlog = 4096 # Netty backlog for
connections, should suffice, increase if failures happen
![Page 32: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/32.jpg)
Monitoring
• Application level monitoring• Listener Actors contain Custom JMX MBeans• Direct passthrough of msg or translation
class SomeActor(listener: Option[ActorRef] = None) extends Actor { def receive = { case msg:Msg=> { //handle the msg … // passthrough msg listener.foreach { a => a ! msg } } …
class SystemMonitor(reg: BeanRegistry) extends Actor { def receive = { case msg: SomeInterestingEvent => someJMXBean.processMessage(msg) }}
![Page 33: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/33.jpg)
Testing with TestKit
class TestKitUsageSpec extends WordSpec with BeforeAndAfterAll with ShouldMatchers with TestKit { override protected def afterAll(): scala.Unit = { stopTestActor echoRef.stop} "An EchoActor" should { "Respond with the same message it receives" in { within(100 millis) { echoRef ! "test" expectMsg("test") } } }
![Page 34: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/34.jpg)
Testing one-way Actor Output
trait ReplyAfterProcessing extends Actor { abstract override def receive = { super.receive andThen { case msg => self.reply_?(msg) } }}
class SomeSpec extends WordSpec { "A oneway actor" should { "Tell me when it's finished "in { val test = Actor.actorOf(new SomeActor() with ReplyAfterProcessing) var reply = test !! (message, 1000) if(reply.isEmpty) fail("some message") } }}
![Page 35: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/35.jpg)
Microkernel deployment
• sbt dist (akka-sbt-plugin)• Creates dist directory• init.d scripts
![Page 36: Real world Scala hAkking NLJUG JFall 2011](https://reader035.vdocuments.us/reader035/viewer/2022062512/5549ab34b4c90507608b57ef/html5/thumbnails/36.jpg)
Questions?