Sociable Software(or “how you really already knowhow to make concurrency for multicoreeasy to work with”)

http://kamaelia.sourceforge.net/Developers/

Michael Sparksms@cerenity.org

BBC Research & Innovation

This presentation aims to show people that they already knowhow to deal with concurrency.

It argues that if we have the tools for large scale concurrency(mashups) and small scale (hardware) that midrange (normalapps) can be done in a similar way, using existing tools.

This is done by showing useful systems that have been producedin this manner using existing tools.

ie from existing practice, not theory

http://kamaelia.sourceforge.net/Developers/ ms@cerenity.org

Note added after barcamp

During the actual presentation I also talked about Kamaeliaprojects created by novice programmers of varying ability whichshow high levels of concurrency.

These include: previewing PVR content on mobiles, multicastisland joining, as-live streaming using bit torrent, Open GL baseduser interfaces & integration, seaside style webserving, speexbased secure phone, IRC/IM systems, a shakespeare scriptplayer, and games tools.

Other systems created include Atom/RSS routing, memcachedintegration, P2P whiteboarding (with audio + mixing), gesturerecognition, presentation tools, a kids development environment,topology visualisation tools, database modelling etc.

http://kamaelia.sourceforge.net/Developers/ ms@cerenity.org

Note added after barcamp

Sociable Software

Core idea : “building systems that are made of many software systems that are sociable with each other”

(yes, this project did start at work, but I am primarily working on it in my own time at present due to a resource crunch :)

Sociable Software

ie it's comprised of systems that

talk to each other.

ie are sociable.

Comfortable examples

Web Browser

Web Serverresponse

request

Web Browser

Web Server

request

RSS/Atom Feed

More sociable:

Web Browser

Web Server

request

RSS/Atom Feed

More sociable:

Kinda like your facebook mini-feed

Web Browser

General principle

Web Server

RSS/Atom Feed

Web Server

RSS/Atom Feed

Web Server

RSS/Atom Feed

Mash ups(next logical step)

Basic client side mashup

Web Server

Atom Feed

Web Server

Web API

Web Server

Someclient

RSS Feed

Basic client side mashup

Web Server

Atom Feed

Web Server

Web API

Web Server

Someclient

RSS Feed

Example: Google Ads

Basic client side mashup

Web Server

Atom Feed

Web Server

Web API

Web Server

Someclient

RSS Feed

Example: Google Ads Or any similar sort of system ie using Javascript for transcludes

Mashup as a service

Someclient

Web Server

Atom Feed

Web Server

Web API

Web Server

Web Server

RSS Feed

Mashup as a service

Someclient

Web Server

Atom Feed

Web Server

Web API

Web Server

Web Server

RSS Feed

Example: gmap-pedometer

Someclient

Web Server

Atom Feed

Web Server

Web API

Web Server

Web Server

RSS Feed

Service

More generally

Go Crazy(next logical step)

Web Server

Web Server

Web Server

RSS Feed

Web Server

Atom Feed

Web Server

Web Server

RSS Feed

Web Server

Web Server

RSS Feed

Web API Web API

Web API

Web API Web API

Web ServerWeb API

Web Server

Service

Web Server

Service

`

Atom Feed

Do you really think of it that way?

Got that?(pretty common stuff yes?)

Let's look at something completely

different.

Formal verification of asynchronous hardware

(Rainbow project, Manchester Uni. ~10 years old)

Doesn't really look so very different does it?

Even complex examples look similar.

Why?

They're comprised of systems that

talk to each other.

So what?

Both represent concurrent systems.

One is very high level.

One is incredibly low level.

The low level has the best tools and builds on decades of experience of making things simpler.

They've made their lives easier

green Pipeline() = Random() >> buffer(1) >> Invert() >> buffer(1) >> Invert() >> buffer(1) >> sinkend

green Substituter() = chan thevalue,istwelve end function test(value x:integer) : boolean = x==12 end Random() >> thevalue >> test() >> istwelve$ thevalue >> if istwelve then sink$ source 0 >> else {} end_if >> sinkend

Um?green Substituter() = chan thevalue,istwelve end function test(value x:integer) : boolean = x==12 end Random() >> thevalue >> test() >> istwelve$ thevalue >> if istwelve then sink$ source 0 >> else {} end_if >> sinkend

Um?

That looks like software!

green Substituter() = chan thevalue,istwelve end function test(value x:integer) : boolean = x==12 end Random() >> thevalue >> test() >> istwelve$ thevalue >> if istwelve then sink$ source 0 >> else {} end_if >> sinkend

Can we write software that way

and still have it perform well,

and gain the same benefits?

YES!

Examples

Examples

I am only going to show you non-trivial examples

For more trivial examples, please look at the slides uploaded to slideshare.net/kamaelian and also the Kamaelia cookbook.

Examples

Email Greylisting Server: Kamaelia Grey

Specifically written to deal with the ever growing spam problem I was facing

Has eradicated my spam problem on my home server.

Examples

Email Greylisting Server:

TCPServer

GreylistServer

Selector

outbox

inbox

outbox

inbox

Examples

Email Greylisting Server: New Connection

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

GreyListingPolicy

outbox

inbox

outbox

inbox

inbox

inboxoutbox

outbox

inbox

inbox

inboxoutbox

inboxoutbox

outbox

Examples

Email Greylisting Server: Many Connections

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

ConnectedSocketAdapter

ConnectedSocketAdapter

GreyListingPolicy

GreyListingPolicy

GreyListingPolicy

Examples

Email Greylisting Core Code:class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] ... port = config["port"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"])

class protocol(GreyListingPolicy): servername = config["servername"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] ...

Examples

Email Greylisting Core Code:class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] ... port = config["port"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"])

class protocol(GreyListingPolicy): servername = config["servername"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] ...

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

GreyListingPolicy

Examples

Email Greylisting Core Code:class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] ... port = config["port"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"])

class protocol(GreyListingPolicy): servername = config["servername"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] ...

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

GreyListingPolicy

Examples

Email Greylisting Core Code:class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] ... port = config["port"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"])

class protocol(GreyListingPolicy): servername = config["servername"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] ...

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

GreyListingPolicy

Examples

Email Greylisting Core Code:class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] ... port = config["port"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"])

class protocol(GreyListingPolicy): servername = config["servername"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] ...

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

GreyListingPolicy

Examples

Email Greylisting Core Code:class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] ... port = config["port"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"])

class protocol(GreyListingPolicy): servername = config["servername"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] ...

TCPServer

GreylistServer

Selector

ConnectedSocketAdapter

GreyListingPolicy

Slightly less obvious than normal, but these functions together form a factory.

TCPServer

GreylistServer

Selector

outbox

inbox

outbox

inbox

TCPServer

GreylistServer

Selector

Slightly less obvious than normal, but these functions together form a factory... so as to create new things to handle connections:

ConnectedSocketAdapter

GreyListingPolicy

TCPServer

GreylistServer

Selector

Slightly less obvious than normal, but these functions together form a factory... so as to create new things to handle connections:

ConnectedSocketAdapter

ConnectedSocketAdapter

GreyListingPolicy

GreyListingPolicy

TCPServer

GreylistServer

Selector

Slightly less obvious than normal, but these functions together form a factory... so as to create new things to handle connections:

ConnectedSocketAdapter

ConnectedSocketAdapter

ConnectedSocketAdapter

GreyListingPolicy

GreyListingPolicy

GreyListingPolicy

Examples

Kamaelia ER Modeller

Examples

Kamaelia ER Modeller

Intended for use in designing

databases. Created last

year because I had an

urgent need.

About a ½ day hack.

Examples

Kamaelia ER Modeller : Simple Example

entity missionagententity person(missionagent)entity team(missionagent)

entity missionitem: simpleattributes visible

entity activemission

relation participatesin(activemission,missionagent)relation creates(missionagent,missionitem)

Examples

Kamaelia ER Modeller : More Complex Example

entity missionagententity person(missionagent): simpleattributes nameentity team(missionagent): simpleattributes teamnameentity missionentity activemissionrelation participatesin(activemission,missionagent)entity missionitem: simpleattributes visiblerelation creates(missionagent,missionitem)relation updates(missionagent,missionitem)entity textfrag(atomicitem)entity externalurl(atomicitem)...

Examples

Kamaelia ER Modeller : Internals

Backplane(“TOPOLOGY”)

Pipeline

Console Reader

PublishTo(“TOPOLOGY”)

Pipeline

FileReader

PublishTo(“TOPOLOGY”)

ERParser

ERModel2Visualiser

Pipeline

ConsoleEchoer

SubscribeTo(“TOPOLOGY”)

ERVisualiser

lines_to_tokenlists

Examples

Kamaelia ER Modeller : Internals

Backplane(“TOPOLOGY”)

Pipeline

Console Reader

PublishTo(“TOPOLOGY”)

Pipeline

FileReader

PublishTo(“TOPOLOGY”)

ERParser

ERModel2Visualiser

Pipeline

ConsoleEchoer

SubscribeTo(“TOPOLOGY”)

ERVisualiser

lines_to_tokenlists

Backplane("TOPOLOGY").activate()

Examples

Kamaelia ER Modeller : Internals

Backplane(“TOPOLOGY”)

Pipeline

Console Reader

PublishTo(“TOPOLOGY”)

Pipeline

FileReader

PublishTo(“TOPOLOGY”)

ERParser

ERModel2Visualiser

Pipeline

ConsoleEchoer

SubscribeTo(“TOPOLOGY”)

ERVisualiser

lines_to_tokenlists

Pipeline( ConsoleReader(">>> "), PublishTo("TOPOLOGY"), ).activate()

Examples

Kamaelia ER Modeller : Internals

Backplane(“TOPOLOGY”)

Pipeline

Console Reader

PublishTo(“TOPOLOGY”)

Pipeline

ConsoleEchoer

SubscribeTo(“TOPOLOGY”)

ERVisualiser

lines_to_tokenlists

Pipeline

FileReader

PublishTo(“TOPOLOGY”)

ERParser

ERModel2Visualiser

Pipeline( ReadFileAdaptor(sys.argv[1]), ERParser(), ERModel2Visualiser(), PublishTo("TOPOLOGY"), ).activate()

Examples

Kamaelia ER Modeller : Internals

Backplane(“TOPOLOGY”)

Pipeline

Console Reader

PublishTo(“TOPOLOGY”)

Pipeline

FileReader

PublishTo(“TOPOLOGY”)

ERParser

ERModel2Visualiser

Pipeline

ConsoleEchoer

SubscribeTo(“TOPOLOGY”)

ERVisualiser

lines_to_tokenlists

Pipeline( SubscribeTo("TOPOLOGY"), lines_to_tokenlists(), ERVisualiser( screensize = (1024,768), fullscreen = True ), ConsoleEchoer(), ).run()

Examples

Kamaelia ER Modeller : Internals

Backplane(“TOPOLOGY”)

Pipeline

Console Reader

PublishTo(“TOPOLOGY”)

Pipeline

FileReader

PublishTo(“TOPOLOGY”)

ERParser

ERModel2Visualiser

Pipeline

ConsoleEchoer

SubscribeTo(“TOPOLOGY”)

ERVisualiser

lines_to_tokenlists

Backplane("TOPOLOGY").activate()

Pipeline( ConsoleReader(">>> "), PublishTo("TOPOLOGY"),).activate()

Pipeline( ReadFileAdaptor(sys.argv[1]), ERParser(), ERModel2Visualiser(), PublishTo("TOPOLOGY"),).activate()

Pipeline( SubscribeTo("TOPOLOGY"), lines_to_tokenlists(), ERVisualiser(screensize = (1024,768), fullscreen = True), ConsoleEchoer(),).run()

Examples

Kamaelia Macro

Examples

Kamaelia Macro

It records and transcodes what is broadcast over DTT for future viewing.

Examples

Kamaelia Macro

It records and transcodes what is broadcast over DTT for future viewing.

     17A.  ­ (1) The making in domestic premises for private and domestic use of a recording of a broadcast solely for the purpose of enabling it to be viewed or listened to at a more convenient time does not infringe any right conferred by Part 2 in relation to a performance or recording included in the broadcast.

­­ Timeshifting as defined by copyright

Examples

Kamaelia Macro

Examples

Kamaelia Macro: Top Level Schematic DVB Multiplex

outbox

Channel Transcoderinbox

DVB  Demuxer  

BBC_ONEinbox

Channel Transcoderinbox

Channel Transcoderinbox

Channel Transcoder

Channel Transcoder

Channel Transcoder

inbox

inbox

inbox

BBC_TWO

BBC_THREE

BBC_FOUR

CBEEBIES

CBBC

Examples

Kamaelia Macro: Transcode progs on a Channel

Channel Transcoder

Splitter

inbox

_eit_

inbox

ProgChangeDetector

outbox

inbox

RestartableTranscoder

transcoder_factory

inbox

nextoutbox

Examples

Kamaelia Macro: Finding Programme Junctions

EIT Parsing

DVB PacketReconstructor

inboxinbox

Now NextParser

outbox

inbox

Now Nextservice filter

Now NextChanges

outbox

inbox

outbox

inbox

outbox

outbox

Examples

Kamaelia Macro: Programme Transcoder

Programme Transcoder

(a pipeline)

Pipethrough(“mencoder ­o file options...”)

inbox control

signal

inbox

control

_transcodingcomplete

_stop

signal

How?

What's inside the box?

The following examples are largely simplified, simply because slides are a terrible way to show code.

I've generally removed shutdown code.

How?

What's inside the box?class ConsoleReader(threadedcomponent): def main(self): while 1: line = raw_input(">>") line = line + self.eol self.send(line, "outbox")

How?

What's inside the box?class ConsoleReader(threadedcomponent): def main(self): while 1: line = raw_input(">>") line = line + self.eol self.send(line, "outbox")

See – we're saying this component runs inside it's own thread. This is all we have to write for it to work.

How?

What's inside the box?class ConsoleReader(threadedcomponent): def main(self): while 1: line = raw_input(">>") line = line + self.eol self.send(line, "outbox")

As you can see here, a component communicates with other components by sending things to outboxes.

How?

What's inside the box?class ConsoleEchoer(component): def main(self): while not self.shutdown(): while self.dataReady("inbox"): data = self.recv("inbox") sys.stdout.write(str(data)) sys.stdout.flush() yield 1

Whereas here, you can see that they listen for data in their inboxes. and then do stuff based on that!

How?

What's inside the box?class ConsoleEchoer(component): def main(self): while not self.shutdown(): while self.dataReady("inbox"): data = self.recv("inbox") sys.stdout.write(str(data)) sys.stdout.flush() yield 1

This one however is a generator based component. This runs in the same thread as all other generators, making this code equivalent to a state machine. Note the baseclass does not say threaded or generator.

How?

What's inside the box?class ConsoleEchoer(component): def main(self): while not self.shutdown(): while self.dataReady("inbox"): data = self.recv("inbox") sys.stdout.write(str(data)) sys.stdout.flush() yield 1

This yield keyword is what makes this method a generator. For now, you can assume it acts like a “return back here when next called” thing. It's NOT really the same at all as a ruby yield!

And that's the basic idea.

On the website there's a mini axon tutorial and a guide to writing components, and a few magazine articles as well which should get you started.

There are also more tutorials on slideshare

Please also ask on #kamaelia on freenode.

Thank you:-)