event infrastructure alasdair allan university of exeter alasdair allan university of exeter robert...

Event InfrastructureAlasdair AllanUniversity of Exeter

Alasdair Allan

University of Exeter

Robert R. WhiteLos Alamos National Laboratories

The eSTAR/TALONS Interconnect

VOEvent Workshop II, Tucson

2

The eSTAR concept

• It’s an ad-hoc peer-to-peer network of heterogeneous resources utilising a collaborative agent paradigm for decision making

• Treat telescopes and databases in a similar manner, both being made available on an observational grid

• Disguise complexity, users hate complexity


3

eSTAR in a nutshell


4

What is an agent anyway?

• An agent is “just software” not magic

Loosely, an agent is a computational entity which:

• Acts on behalf of another entity in an autonomous fashion

• Performs its actions with some level of proactivity and/or responsiveness

• Exhibits some level of the key attributes of learning, co-operation and mobility


5

Multi-agent systems

• A multi-agent system is one that consists of a number of agents, which interact with one-another.

• In the most general case, agents will be acting on behalf of users with different goals and motivations.

• To successfully interact, they will require the ability to cooperate, coordinate, and negotiate with each other, much as people do…


6

Multi-agent systems for eSTAR

• We’ve built the first agent based astronomical system, and it was clearly the correct choice of architecture.

• What we’ve built in eSTAR is a collaborative agent system which schedules telescopes.

• Complicated telescope schedules are built by humans using a multi-pass approach.

• Traveling salesman problem…


7

Peer-to-Peer

• Agents operate in a peer-to-peer manner and can make use of these interconnections between people and data.

• Carrying out intelligent resource discovery could mean that your agent looks to your collaborators agent for data and expertise before it looks to “central” sources.


8

The world is flat…

• The world is small and flat, but it is none the less still very complex.

• Architectures which take account of this are intuitive, and will map well into the real world.

© Terry Pratchett


9

The eSTAR network

© Nik Szymanek

User Agents

Embedded Agent The VO

Embedded Agents

• GRB rapid follow-up programme• Search for exo-planets• Testbed for Robonet-1.0 open time

GatewayService


10

The eSTAR network

© Nik Szymanek

User Agents

Embedded Agent The VO

GatewayService

Embedded Agents

GCNAlertAgent

Broker


11

What we’ve done so far?

• Convenience layer and parser (Perl & C++)– building common cases (including GCN)– different parser methodologies

• Prototype event network live and on sky• Limited (hard-wired logic) brokering• Archiving and persistent storage• RSS test feed(s)

– programmatic, or human readable?


12

Where are we going?

• Event brokering (aggregators)– time critical– non-time critical (data mining?)

• Event archiving and persistent storage– REST interface– persistence of data products

• Search service– REST interface

• Event Feeds– pull, or push?


13

Event feeds

There are two ways to feed events to consumers,

• Pull model (feeds)– polling– e.g. RSS feed

• Push model (forwarding)– via REST, SOAP or vanilla TCP


14

Pull model

• Advantage– under client control

• Disadvantage– as slow as the polling interval

A pull model, e.g. RSS, is never going to work for rapidresponse cases like GRB or transient follow-up.


15

Push model

• Advantage– theoretically faster, depending on architecture

• Disadvantage– administrative nightmare for publisher

A push model, e.g. GCN, is the only way to do rapidresponse work. However it isn’t really optimal, so we’llprobably be stuck using both approaches. Shouldn’toptimise our standard for distribution via RSS model.


16

Aggregators

Why should we aggregate event feeds?

• Consolidation– Do we need to support multiple publishers?

• Removal of duplicate events

• Added semantic content

• Trust issues


17

Storage

Why should we permanently store event messages?

• The data itself will be replicated elsewhere in the VO.

• Why do we care about the original message?

• If we want to search, we have to store.

• However, to me, the best use case for persistent storage is actually event feeds…


18

Atom feed

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"> <title xmlns="http://www.w3.org/2005/Atom">eSTAR/TALONS GCN Feed</title> <author xmlns="http://www.w3.org/2005/Atom"> <name xmlns="http://purl.org/atom/ns#">eSTAR Project</name> <email xmlns="http://purl.org/atom/ns#">[email protected]</email> </author> <link xmlns="http://purl.org/atom/ns#" type="application/atom+xml" rel="self” href="http://www.estar.org.uk/atom.xml"/> <entry xmlns="http://purl.org/atom/ns#" xmlns:default="http://www.w3.org/1999/xhtml"> <title xmlns="http://purl.org/atom/ns#">ivo://gcn.gsfc/hete/396943a</title> <content xmlns="http://purl.org/atom/ns#" mode="xml"> <div xmlns="http://www.w3.org/1999/xhtml"> <![CDATA[<?xml version = '1.0' encoding = 'UTF-8'?> <VOEvent version="1.0" id="ivo://gcn.gsfc/hete/396943a”> . . . </VOEvent> ]]> </div> </content> <author xmlns="http://purl.org/atom/ns#"> <name xmlns="http://purl.org/atom/ns#">GCN</name> <email xmlns="http://purl.org/atom/ns#">[email protected]</email> </author> </entry> . . .</feed>


19

RSS feed

<?xml version="1.0" ?><rss version="2.0"> <channel> <title>VOEvent GCN Notices</title> <language>en</language> <description>RSS feed for GCN notices</description> <link>http://voeventnet.caltech.edu/GCN.html</link> <item> <title>MILAGRO_Source trigger 886-1</title> <description> <p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p> </description> <pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate> <link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link> <author>Scott Barthelmy GCN [email protected]</author> <comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments> </item> . . . </channel></rss>


20

<enclosure>

• Its an optional sub-element of the RSS <item> tag that allows the feed publisher to include a link to a file.

• It has three required attributes. The url=“ ” attribute says where the enclosure is located, length=“ ” says how big it is in bytes, and type=“ ” says what its type is, a standard MIME type. e.g,

<enclosure url=“http://estar.org.uk/event.pl?id=886-1” length=“2440” type=“application/xml+voevent” />

• This is how podcasting works..


21

RSS feed

<?xml version="1.0" ?><rss version="2.0"> <channel> <title>VOEvent GCN Notices</title> <language>en</language> <description>RSS feed for GCN notices</description> <link>http://voeventnet.caltech.edu/GCN.html</link> <item> <title>MILAGRO_Source trigger 886-1</title> <description> <p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p> </description> <pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate> <link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link> <author>Scott Barthelmy GCN [email protected]</author> <comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments> </item> . . . </channel></rss>


22

RSS feed

<?xml version="1.0" ?><rss version="2.0"> <channel> <title>VOEvent GCN Notices</title> <language>en</language> <description>RSS feed for GCN notices</description> <link>http://voeventnet.caltech.edu/GCN.html</link> <item> <title>MILAGRO_Source trigger 886-1</title> <description> <p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p> </description> <pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate> <link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link> <author>Scott Barthelmy GCN [email protected]</author> <comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments> <enclosure url=“http://estar.org.uk/event.pl?id=886-1” length=“2440” type=“application/xml+voevent” /> </item> . . . </channel></rss>


23

Unanswered questions

We have our document standard, so at this stage there

should be only two issues outstanding,

• Message passing protocol(s)– How the documents are passed

• Transport standard(s?)– How the documents are carried


24

Protocol work

The recent work on interoperability between eSTAR and

TALONS has show the importance of,

• ACK messages

• IAMALIVE messages

GatewayService

eSTARBroker


25

Should we mandate transport?

• Why? Nobody thought about RFC 1149 when IP datagram packets were standardised…

• See http://www.blug.linux.no/rfc1149/

http://www.blug.linux.no/rfc1149/index.html






26

Authentication

• Transport layer issue?

• Format issue?


27

Authorisation

• Transport layer issue?


28

Things to think about…

• Standards never survive widespread contact with users intact. At least no good standard does…

• If people twist and pull the standard to do something differently than we designed it to do, that’s our fault.

• The users are always right. No, seriously…

event infrastructure alasdair allan university of exeter alasdair allan university of exeter robert...

Documents

complexityvoevent workshop

mobilityvoevent workshop

nutshellvoevent workshop

collaborators agent

people dovoevent workshop

vanilla tcpvoevent workshop

open timevoevent workshop

collaborative agent