Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

EmbracingConcurrency

for Fun, Utility & Simpler Code

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Hardware finally going massively concurrent ...

.... PS3, high end servers, trickling down to desktops, laptops)

Why?

“many hands make lightwork” but Viewed as Hard... do we just have crap tools?

“And one language to inthe darkness bind them”... can just we REALLY abandon 50 years of code for Erlang, Haskelland occam?

Opportunity!

Problems

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Missing Something?Fundamental Control Structures... in imperative languages number greater than 3! (despite what you get taught...!)

Control Structure Traditional Abstraction Biggest Pain Points

Sequence Function Global VarSelection Function Global VarIteration Function Global VarParallel Thread Shared Data

Usually Skipped Lost or duplicate update are most common bugsThis slide raises the point that

teaching of imperative languages often misses out concurrency as afundamental construct, leaving a

conceptual gap for most developers

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Desktop

APPS

Network

Novice

3rd Party

Media

gsoc

trainee

Regarding using concurrency, what

sort of applications are we talking about

here?

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Desktop

APPS

Network

Novice

3rd Party

P2P Whiteboard

ER DB Modeller Kids

Programming(logo) Simple

Games

Speak 'n Write

UGCBackend

Transcoder

Think backendneeded foryoutube/flickrtype systems

Media

Compose

Shot ChangeDetection

MobileReframing

DVB

Macro“record

everything”

Podcasts

Email &SpamSMTP

GreylistingPop3Proxy ClientSide

Spam Tools

IRCWeb

Serving

gsoc

trainee

AIM

AWS(Amazon)

SednaXMLDBXMPP

pubsubQt

Gtkmicroblogging

MiniAxonScriptReaderMediaPreviewon MobileReliableMulticast

P2P RadioTorrent3D SystemsRealtime MusicPaint AppP2P Web ServerSecure “phone”Social Net Vis...

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Core Approach: Concurrent things with comms points Generally send messages Keep data private, don't share

inbox

control

...

outbox

signal

...

inbox

control

...

outbox

signal

...

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

But I must share data?Use Software Transactional Memoryie version control for variables.

1. Check out the collection of values you wish to work on2. Change them3. Check them back in 4. If conflict/clash, go back to 1

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Perspectives in APIs! (1/2)

inbox

control

...

outbox

signal

...

1st Person - I change my state

2nd Person – YOUwant to me to dosomething(you sendme a message)

3rd Person –Bla shoulddo something(I send a message)

If you have concurrency it becomes natural to think in terms of 1 st 2nd and 3rd person. This affects an API's structure, and can be vital for understanding it!

This is one we've found that makes sense

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

inbox

control

...

outbox

signal

...

private real methods

Messagesfrom publicinboxes

Messages sentto public outboxes

Perspectives in APIs! (2/2)

Also, thinkabout stdin

Also, thinkabout stdout

If you have concurrency it becomes natural to think in terms of 1 st 2nd and 3rd person. This affects an API's structure, and can be vital for understanding it!

This is one we've found that makes sense

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

inbox

control

...

private real methods

Messagesfrom publicinboxes

Actor SystemsDistinction can be unclear,potential source of ambiguity*

No outbox conceptPossible issues withrate limiting*Hardcodes recipientin the sender

*system dependent issue

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Advantages of outboxes

inbox

control

...

outbox

signal

...

No hardcoding of recipientallows: - Late Binding - Dynamic rewiring

Concurrency Patterns as Reusable Code ... a concurrency DSL

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

A Core Concurrency DSLPipeline(A,B,C)Graphline(A=A,B=B, C=C, linkages = {})Tpipe(cond, C)Seq(A,B,C), PAR(), ALT()Backplane(“name”), PublishTo(“name”), SubscribeTo(“name”)Carousel(...)PureTransformer(...)StatefulTransformer(...)PureServer(...)MessageDemuxer(...)Source(*messages)NullSink

Some of these are work in progress –

they've been identified as useful, but not implemented as chassis, yet

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Pipeline ExamplePipeline( MyGamesEventsComponent(up="p", down="l", left="a", right="s"), BasicSprite("cat.png", name = "cat", border=40),).activate() MyGames

EventsComponent

BasicSprite

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Graphline ExampleGraphline( NEXT = Button(...), PREVIOUS = Button(...), FIRST = Button(...), LAST = Button(...), CHOOSER = Chooser(...), IMAGE = Image(...), ...).run()

FIRST(button)

Chooser

LAST(button)

PREVIOUS(button)

NEXT(button)

Image

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Server Example

MainServer Core

Remote User

Protocol Handler Factory

Socket handler

Protocol handler

datato

user

datafromuser

Created at runtime to handle the connection

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Server Example

MainServer Core

Protocol Handler Factory

You therefore need to provide this bit.

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Server Examplefrom Kamaelia.Chassis.ConnectedServer import ServerCorefrom Kamaelia.Util.PureTransformer import PureTransformer

def greeter(*argv, **argd): return PureTransformer(lambda x: "hello" +x)

class GreeterServer(ServerCore): protocol=greeter port=1601

GreeterServer().run()

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Backplane Example# Streaming Server for raw DVB of Radio 1 Backplane(“Radio”).activate()

Pipeline( DVB_Multiplex(850.16, [6210], feparams), # RADIO ONE PublishTo("RADIO"),).activate()

def radio(*argv,**argd): return SubscribeTo(“RADIO”)

ServerCore(protocol=radio, port=1600).run()

Michael SparksBBC R&D, http://www.kamaelia.org/Home.html

Thank you for listening!

If you have questions, grab me later :-)

Kamaelia is open source

Kamaelia Tutorial (from Europython 09) : - http://tinyurl.com/kamaelia