skyalert: rapid transients for you and your robot

1

Skyalert: Rapid Transients for you and your Robot

Roy WilliamsLIGO Laboratory, Caltech

Andrew Drake, Ashish MahabalAstronomy, Caltech

Supported by US National Science Foundation OCI-0915473

2

Skyalert is ….A clearinghouse and repository of information about astronomical transients, each described by a collection of VOEvent packets that may be multiply authored. The components of Skyalert are:• A web-based event broker, allowing subscription so that information about transients can

be delivered to users and their telescopes immediately upon receipt.• A web-based authoring system, so that authenticated users can inject events direct from

automated discovery pipelines, or fill in web forms, that may be delivered rapidly to others.

• An event repository, storing all events that come through the broker, and allowing bulk queries and drill-down.

• A ‘click or code’ paradigm that allows people web based access and machines web service access.

• A way to browse recent and past transients, as tables, multi-layered web pages, or with popular astronomical software.

• A development platform for building real-time decision rules about transients, and for mining the repository.

• Open-source software to allow local implementations as well as the web-based application.

http://lib.skyalert.org/skyalert/Guide2Skyalert-1.3.pdf


http://www.skyalert.org/s/sampleEvent.html

3

Installations

• Production: Skyalert.org at Caltech• Test: Betelgeuse.ligo.caltech.edu:8000• IUCAA, India• LSST, Tucson, Arizona• AMON, Penn State, Pennsylvania• GAIA, Cambridge UK

http://lib.skyalert.org/skyalert/



4

MicrolensingOptical transients

Radio transientsX-ray transients

Gamma transients

Follow-up Scheduler

TelescopeTelescope

Telescope

Authors Subscribers

annotation from archives

Events and annotation disseminated to subscribers

in real time with intelligence

skyalert.org

AstronomersAmateursStudents

Skyalerthttp://skyalert.org

GCN Broker

http://skyalert.org/


5

VOEvent is the common language

http://ivoa.net/Documents/VOEvent/

http://hotwireduniverse.org



http://hotwireduniverse.org/





6

<voe:VOEvent xsi:schemaLocation="http://www.ivoa.net/xml/VOEvent/v2.0 /Users/roywilliams/Documents/VOEventLib/schema/VOEvent-v2.0.xsd" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:voe="http://www.ivoa.net/xml/VOEvent/v2.0" version="2.0" ivorn="ivo://ldas-pcdev1.ligo.caltech.edu/VOEvent#a8" role="test">

<Description>Candidate galaxies for LIGO trigger event</Description> <Reference uri="http://ldas-pcdev1.ligo.caltech.edju/VOEventRepository/1"/>

<Citations> <EventIVORN cite="followup">ivo://ldas-pcdev1.ligo.caltech.edu/VOEvent#a7</EventIVORN> </Citations>

<Who> <Author> <title>LSCVirgo</title> <contactName>Ed Daw</contactName> <contactEmail>[email protected]</contactEmail> <contactPhone>+44 114 268 6895</contactPhone> </Author> </Who> <What> <Param name="EVETut_ID" value="012345678" ucd="meta.id"> <Description>use the gps second as an event ID </Description> </Param> <Param name="SNR" value="2.4" ucd="stat.snr"> <Description>Significance of the trigger</Description> </Param> <Param name="UTC" value="2005-04-15T23:59:59" ucd="time.epoch"> <Description>Time of the trigger</Description> </Param> <Param name="NO_OF_GALAXIES" value="3" ucd="meta.number"> <Description>Number of candidate source galaxies in this VOEvent</Description> </Param> <Table> <Description>Candidate sources for the LSC/Virgo trigger</Description> <Field name="Source rank" ucd="meta.code"> <Description>this integer is 1 for the most likely source, 2 for 2nd, etc</Description> </Field> <Field name="GWGC_CATALOGUE_NUMBER" ucd="meta.number"> <Description>this is the integer ID of the galaxy in Darren's catalogue</Description>

VOEvent

• Who – author’s provenance• What – author’s data packet• WhereWhen – targeting in spacetime• Why – scientific characterization• Citation – related events• IVORN – unique identifier


http://hotwireduniverse.org



http://hotwireduniverse.org/





7

VOEvent validation

Choose herefor validation

http://www.skyalert.org/submit/



8

VOEventLib

• Python Library for VOEvent– Read, modify, create,

write VOEvents– lib.skyalert.org/

VOEventLib– Code built

automatically from XML schema• GenerateDS package• Thanks Dave Kuhlman!

from VOEventLib.VOEvent import *from VOEventLib.Vutil import *import sys# get filename with event to be modified

# parse the event from the file namev = parse(filename)

# set the ivorn to something elsev.set_ivorn('ivo://silly/billy#iuw6e7F72ufh')

# set the authorv.get_Who().get_Author().set_contactName(['Mickey Mouse'])

# look for a specific param in the event to changeparam = findParam(v, '', 'magnitude')if param: val = paramValue(param) param.set_value("16.45")

xml = stringVOEvent(v)print xml

http://lib.skyalert.org/VOEventLib/



9

Who uses VOEvent

• IAU/CBAT, AAVSO• GAIA• CRTS, CSS asteroids, MOA, OGLE• SWIFT, Fermi, Integral, Agile, etc

(GCN translation)

• LIGO, LOFAR• LSST?• amateurs (DC3.com)

10

Event timeline

click event for drilldown

http://www.skyalert.org/

http://www.skyalert.org/


11

Worldwide Telescopeview of all recent events

CLICK

12

Skyalert Key Concepts• Identifier

• IVORN• Stream (class) IVORN inside event (instance) IVORN

• Stream• Event semantics defined by author

– What does Burst_SOD mean in the Fermi stream?• Like a data dictionary, a schema, common metadata

• Portfolio• Event + associated data

– As collection of VOEvents with the observation first

• Alert/Select• Define what is ‘interesting’ as Python expression on Params

– eg CRTS[“magnitude”] < 17– Get message, run annotator, run telescope

• Actually … everything is an Alert

?

FermiCRTS3PIoftheSKY

13

Concept: Identifier== IVORN

• Can identify a stream:• ivo://moa.massey/transient_alerts

• Can identify an event from that stream:• ivo://moa.massey/transient_alerts#gb10-R-8-16933_1

The event IVORN contains the stream IVORN

http://www.ivoa.net/Documents/latest/IDs.html



14

Concept: Stream as Event Template

Event author defines parameters and meaning in advance.

stream event

made byperson in daytime

made byrobot system at night

15

Stream DefinitionName, IVORN, ownerDescription

Overview Template

See events, delete tests

Parameters that can be used in the overview

Sample event

Owner view Public view

http://www.skyalert.org/streams/54/



16

Skyalert Stream Registry... will be VO registry

http://www.skyalert.org/streams/



17

Concept: Portfolio

==“data holdings about a transient”Multiple authors

Primary event plus secondary data: follow-up, archives, data-mining results etc

In VOEvent there are <Citation> tags

18

Portfolio skins

OverviewParamsXML

http://www.skyalert.org/events/46556/



19

CRTS portfolio

CRTS observation Later CRTSobservation

Is it in the archivesyes: NED and NVSS

Telescopic followup

Classification asblazar outburst




20

SWIFT portfolio

Notices from the satellite Circulars from the humans




21

What is an Alert?

• Selects “interesting” events– For rapid messaging– For response to query– Default action is to send email, some others:

• addconstellation• alert_email• coincidence• csannotator• post2skyalert• sdssannotator• senddakota

?

22

Annotators

• When triggered, annotator builds event, appends to portfolio

• Annotator can be local or remote

• Example• For each event, find

nearest galaxy in SDSS

http://www.skyalert.org/s/annotator.html


23

Every Alert is a FeedAtom and RSS

http://pipes.yahoo.com/pipes/


24

Skyalert Overview

GCNEmailJabberDakota etc

MessagingEmailJabberDakota etc

Authoring

= http POST

Annotation

Replication

BrowsingJSON servicesFeeds

Stre

ams

Events

Portfolio

?

??

Alert

25

Users, Groups, Streams Auth and Auth

Users registerGet email & respond

Each stream has User who is owner

Users can join Groups

Groups can have permission to view streams

26

Tutorials

• Tut1: Stream, Sample event, Authoring• Tut2: Authoring with Code• Tut3: Alert, Annotation• Tut4: Mining

Please use test machine https://betelgeuse.ligo.caltech.edu:8000/

27

Tut1: StreamLook at “Test” stream

Anonymous and Logged in

Play with template

Look at bottom for “Sample Event”

https://betelgeuse.ligo.caltech.edu:8000/streams/54/


28

Tut1: Sample

Copy sample and Edit

Change IVORNivorn="ivo://testing/test#5551"Change RA, DecChange ISOtimeChange Params

Go to Authoring page

https://betelgeuse.ligo.caltech.edu:8000/streams/sampleEvent/54/

Keep this on your desktop


29

Tut1: Validation and Authoring

Select “Validate XML”Click to Go

Now authoring …Enter stream name “Test”Enter username/passClick to Go

https://betelgeuse.ligo.caltech.edu:8000/submit/


30

Tut1: Authoring

Event successfully saved

Portfolio visible

31

Tut2: Authoring by code

• Very simple code– http gets through firewalls– Standard Python/Perl etc

• Can also author with – Jabber, Dakota, Email

http://www.skyalert.org/s/authorcode.html

import urllibdict = {}

# the server that will handle the submit requesturl = "https://betelgeuse.ligo.caltech.edu:8000/submit/"

# choose 'dryrun' for validation and 'author' for authoringdict['checker'] = 'dryrun'

# for command line, we want plain text output, not HTMLdict['plainResponse'] = 'on'

# uncomment these for authoring#dict['checker'] = 'author'

# Skyalert username and password#dict['username'] = username#dict['password'] = password

# This is the short name for the stream,# must match credentials and event!#dict['streamName'] = 'myEventStream'

# Should alerts be run once the event is ingested?#dict['doRules'] = 'on'

# open a file for the XMLdict['xmlText'] = open('sample.xml').read()

# Now send it off and print the resultparams = urllib.urlencode(dict)f = urllib.urlopen(url, params)result = f.read()print result

Change serverPut in username/passwordMake XML file from desktop

must change IVORN


32

Log in to build your own alerts and feeds

Select the “Test” stream

Tut3: Building an Alerthttps://betelgeuse.ligo.caltech.edu:8000/rules/newrule/

?

https://betelgeuse.ligo.caltech.edu:8000/rules/newrule/

33

Tut3: Building an Alert

Select the “Test” streamGive it a nameChoose action as csannotator

Make trigger expressionCan use red dots

belowOr type Python

Click Save

https://betelgeuse.ligo.caltech.edu:8000/rules/337/

?


34

Tut3: Authoring with Consequences

Run with Alerts

Have changed RA and Dec of event to194.95 and 27.98

35

Tut3: Annotator Ran

Alerts run on Test events

Annotator fetches data from VO archives

36

Tut3: Discover Coma Cluster

Annotator (VO archives) reveals Coma Cluster from NED

https://betelgeuse.ligo.caltech.edu:8000/events/49663/


37

Tut4: Mining the Events

Alert is a query

Use JSON to get query result-- page/resumption

Then JSON to get portfolio detail

38

Tut4: Select• List the portfolios that satisfy an

alert with JSON return: /jtable/{ "rule": "285", "query": "trigger expression", "next": resumption URL, "list": [ [ "http://....", "ivo://....", "2011-09-19 02:12:23" ],

from django.utils import simplejsonimport urllib

sk = ‘http://skyalert.org’

# CRTS and CRTSCircular["First"]["eventClass"] == "Supernova"query = "285"

jtableurl = "%s/events/jtable/%s/" % (sk, query)

while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for triple in list: ivorn = triple[1] print ivorn

http://www.skyalert.org/events/jtable/285/


39

Tut4: Resolve• Resolve an IVORN to the

portfolio: /resolve/ { "url": "http://skyalert.org/events/50425", "portfolio": { "ivo://.....": { "Group1": { "Param1": "...", "Param2": "….",

# get magnitude from a CRTS eventsk = "https://betelgeuse.ligo.caltech.edu:8000"dict = {‘ivorn’:ivorn, ‘JSON’:’on’}params = urllib.urlencode(dict)resolveurl = "%s/resolve/" % skjsondetail = urllib.urlopen(resolveurl, params).read()pydetail = simplejson.loads(jsondetail)primaryevent = pydetail["portfolio"][ivorn]mag = primaryevent["FirstDetectionparams"]["magnitude"]

http://www.skyalert.org/resolve/


40

Tut4: Mining the Eventsfrom django.utils import simplejsonimport urllib

#sk = "http://skyalert.org"sk = “https://betelgeuse.ligo.caltech.edu:8000”

# CRTS and CRTSCircular["First"]["eventClass"] == "Supernova"query = "285"

jtableurl = "%s/events/jtable/%s/" % (sk, query)

while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for pf in list: ivorn = pf[1]

dict = {} dict['ivorn'] = ivorn dict['JSON'] = 'on' params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) # ivorn of the primary event of the portfolio should be here crtsevent = pydetail["portfolio"][ivorn] mag = crtsevent["FirstDetectionparams"]["magnitude"]

tok = ivorn.split('#') localivorn = tok[1] date = crtsevent[""]["ISOtime"] print localivorn, date, mag

jtableurl = jsonresult["next"] if jtableurl == "finished": break

1107030071174105491 2011-07-03T08:36:01 18.9354001107030091124153498 2011-07-03T08:01:30 18.7563991107031180794128821 2011-07-03T04:39:47 20.1698001107031070764142675 2011-07-03T04:07:39 18.1661001107021180094118167 2011-07-02T10:46:06 18.6014001107021041264109182 2011-07-02T09:31:06 15.8796001107011150104107901 2011-07-01T10:47:36 18.2796991107011120104119909 2011-07-01T10:46:15 17.8748001107010040834139996 2011-07-01T06:17:32 18.4836011107011040674116917 2011-07-01T04:13:08 19.630899

Magnitudes of CRTS supernovae

41

Skyalert Statistics

303 registered

users

61 gmail.com 24 aol.com 15 yahoo.com 7 hotmail.com 6 mail15.com 6 astro.caltech.edu 5 camerabuy.info 5 cacr.caltech.edu 4 caltech.edu 3 lsst.org 2 yahoo.com.br 2 yahoo.co.uk 2 ya.ru

http://webstats.skyalert.org

http://webstats.skyalert.org/

42

Skyalert Statistics2724 emails sent May 18 thru June 18

new full

Which streams generated them?910 SWIFT785 CRTS340 CBAT204 CRTSCircular192 MOA156 CRTS2111 Fermi20 AAVSO6 HST_MCT

Who got them?324 [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

Emails per day

43

Questions?

Please register at skyalert.org

44

Slide 2: http://lib.skyalert.org/skyalert/Guide2Skyalert-1.3.pdfSlide 3: http://lib.skyalert.org/skyalert/Slide 4: http://www.skyalert.orgSlide 5 and 6: http://ivoa.net/Documents/VOEvent/Slide 7: http://www.skyalert.org/submit/Slide 8: http://lib.skyalert.org/VOEventLib/Slide 10: http://www.skyalert.orgSlide 13: http://www.ivoa.net/Documents/latest/IDs.htmlSlide 15: http://www.skyalert.org/streams/54/Slide 16: http://www.skyalert.org/streams/Slide 18: http://www.skyalert.org/events/46556/Slide 19: http://www.skyalert.org/events/33547/Slide 20: http://www.skyalert.org/events/49299/Slide 22: http://www.skyalert.org/s/annotator.htmlSlide 23: http://pipes.yahoo.com/pipes/Slide 27: https://betelgeuse.ligo.caltech.edu:8000/streams/54/Slide 28: https://betelgeuse.ligo.caltech.edu:8000/streams/sampleEvent/54/Slide 29: https://betelgeuse.ligo.caltech.edu:8000/submit/Slide 31: http://www.skyalert.org/s/authorcode.htmlSlide 32: https://betelgeuse.ligo.caltech.edu:8000/rules/newrule/Slide 33: https://betelgeuse.ligo.caltech.edu:8000/rules/337/Slide 36: https://betelgeuse.ligo.caltech.edu:8000/events/49663/Slide 38: http://www.skyalert.org/events/jtable/285/Slide 39: http://www.skyalert.org/resolve/Slide 41: http://webstats.skyalert.org




















https://betelgeuse.ligo.caltech.edu:8000/rules/newrule/





http://webstats.skyalert.org/

skyalert: rapid transients for you and your robot

Documents

skyalert skyalert

webbased event broker

intelligence skyalert

components of skyalert

event repository

webbased application

web forms

voevent import