arizona-noao temporal analysis and response to events...
TRANSCRIPT
Arizona-NOAO Temporal Analysis and Response to Events System
(ANTARES) Tom Matheson
National Optical Astronomy Observatory
Users Committee, NOAO, May 2015 (D2) 1
ANTARES What It Is and Why We Need It
• ANTARES is an astronomical time-domain event broker • A broker is an intermediary who sits between the source
of the alerts and the consumers of the alerts while adding value
• LSST will compare each image with a reference and issue an alert for any 5σ change (brightness or position)
• LSST will generate 1-10 million alerts each night for the ten-year duration of the survey
• We need a system to rapidly winnow that number down to something commensurate with follow-up resources
• Rate and volume will be unprecedented
Users Committee, NOAO, May 2015 (D2) 2
ANTARES Need for a Broker
• One of the primary recommendations from the ‘Spectroscopy in the Era of LSST’ workshop
Users Committee, NOAO, May 2015 (D2) 3
3.1 Software Infrastructure Although time domain science is the primary beneficiary of a broker to sort through the alerts generated by changes in brightness (or position), the cosmological use of SNe Ia, the study of AGNs , and the study of brown dwarf weather all will rely on the ability to recognize interesting events when they happen. For each case, the signal hidden in the noise of one million alerts per night will be different, but they will all require a system to winnow alerts done to the ones they will find of interest. Hidden among those million alerts will be rare and interesting objects, and these will be lost without a working broker. Matheson et al. 2013 (astro-ph 1311.2496)
ANTARES Need for a Broker
• One of the primary recommendations from the NRC report “Optimizing the U.S. Ground-Based Optical and Infrared Astronomy System” (Elmegreen report)
Users Committee, NOAO, May 2015 (D2) 4
Recommendation 4a: The National Science Foundation should help to support the development of event brokers, which should use standard formats and protocols, to maximize LSST transient survey follow-up work.
ANTARES Building on Other Brokers
• Astronomical brokers – Humans – Skyalert (http://skyalert.org/) – PTF (Bloom et al. 2012) – Requires time and information to sort alerts
• LSST problem is one of scale and rate – Multidisciplinary with computer science – Collaboration between NOAO and UA Computer
Science – NSF INSPIRE proposal was successful (IIA-1344024)
Users Committee, NOAO, May 2015 (D2) 5
ANTARES How it Fits in the Transient Ecosystem
• Alert generators find transients and assess validity
• ANTARES characterizes, annotates, and ranks
• Downstream brokers/users classify and follow up
Users Committee, NOAO, May 2015 (D2) 6
ANTARESAnnotation
CharacterizationRanking
Distribution
CRTS LSSTLIGO ZTF
AlternateBrokersAlternate
BrokersAlternateBrokers
Downstream BrokersDownstream
Brokers
Downstream Brokers
Downstream Brokers
ClassificationDistribution
Robotic Telescopes
Investigators
ANTARES Environment
Potential Aggregator of Alerts from Multiple Sources
ALERT GENERATORS: Difference Imaging, Real/Bogus & Moving Object Assessment
Manually Triggered Followup Observing
System Data Flow Info to/from Investigators Feedback
ANTARES Overall Architecture
• Core flow for alerts • Draws on local databases for
annotation • Various stages of filtering • No alerts are lost, just diverted to
other channels • Prototype focuses on ‘rarest of
the rare’ • Open source/open access
Users Committee, NOAO, May 2015 (D2) 7
LSST
Initial Ingest
ExistingCatalogs
1,000 images / night
Thumbnails Updated
CalibrationDatadaily
Camera AlertsUp to 10,000 alerts / image
quarterly
Measurement Recalibration
AstroObject & Alert
Association
AstroObject-Annotated Alerts
Feature Evaluation
Extended Alerts
Alert Characterization
Characterized Alerts
Initial Filtering
Level I Alerts
Advanced Feature Evaluation, Characterization and Filtering
Level II Alerts
1,000 / image
100 / image
1.0 / image
Taxonomy of Characterizations
(curated)
filteredalerts
filteredalerts
FilteringTrees
(curated)
CharacterizationAlgorithms(curated)
Locus-AggregatedAnnotated Alerts
PeriodicQueries
DeriveFeatures
New AstroObjectIdentification
PositionEstimation
MovingObjects(curated)
MovingObject
Identification
Feature Algorithms(curated)
LocusAggregation
ANTARES
Theoretical Predictions
AlgorithmVetting
Professionals,citizen scientists
forking
forking
start at imageacquisition
Other External Alerts
External Alert Ingestion
Theorists
Touchstone(curated)
Alert Throttling
Thumbnail Separation
Combo Assembly
Novel Alerts0.1 / image
"Rarest of the Rare"
Annotated Coalesced
Alerts
filtered andnovel alerts
External Alert
Brokers
Researchersand public
Ultra Characterization & Filtering
30 Dec 2014
Alert Replica Coalescing
Thumbnail Reattachment
black = usgreen = not usblue = catalog by-productsred = alert flow
Annotated Coalesced Alert
Rarest Alerts
AggregatedAstroObject Catalogs
(curated)Aggregated
AstroObject Catalogs(curated)Aggregated
AstroObject Catalogs(curated)
forking
Combo ReplicasSelected Alert Replicas
Combo Feature EvaluationAdvanced Feature Evaluation
Combo Coalescing
ANTARES Annotation Architecture
• Added value is critical • Multiwavelength catalogs,
including LSST itself • Past history of alerts • Other alert sources (e.g.,
LIGO) • Theoretical predictions
Users Committee, NOAO, May 2015 (D2) 8
Databases of astronomical information must be local
ANTARES Filtering Architecture
• Derive features from alert and annotation data
• Use features to characterize (not classify) objects
• Alerts can be forked if more than one possibility
• Filter based on features/characterizations
• As alert numbers drop, more complex processing can occur
Users Committee, NOAO, May 2015 (D2) 9
ANTARES: Progress I
• NSF grant funded Sep 2013 (INSPIRE IIA-1344024) • Weekly meetings to manage development • Astronomy postdoc hired Sep 2014 • 3 CS grad students started Sep 2014
– Faster algorithm testing – Data and algorithm provenance control – Parallel processing experimentation – API developed – Documentation
Users Committee, NOAO, May 2015 (D2) 10
ANTARES: Progress II
• Databases for annotation developed • Can predict Galactic stellar variability • Demonstration prototype operational
– Utilizes major stages of architecture – Modular components can become more complex
Users Committee, NOAO, May 2015 (D2) 11
ANTARES Making Decisions When You Know Nothing
• Some alerts will have no contextual information for annotation
• Kepler provides empirical variability model for Galactic stars
• We can assess variability for 155000+ stars over a wide range of spectral types
• Biased sample, but still the best source
Users Committee, NOAO, May 2015 (D2) 12
12000 10000 8000 6000 4000
65
43
21
0
T eff
log
g
Kepler Sample
T eff
log
g
ANTARES Making Decisions When You Know Nothing
• Distribution of variability for Kepler sample in Q13
• Bright excursions mainly M star flares
• Faint excursions mainly eclipsing binaries
Users Committee, NOAO, May 2015 (D2) 13
−1 0 1 2 3
110
010
000
1000
000
1000
0000
0
Full Kepler Sample
∆mag
Freq
uenc
y
Min. −1.105 Max. 3.335
0.005% 0.05% 99.95% 99.995%−0.276 −0.108 0.152 0.429
ANTARES Making Decisions When You Know Nothing
• As an example, for the LSST reference field (RA 0 DEC 0, l 96 b -60), Besançon model provides distribution of stellar types
• Scale Kepler sample to match Besançon distribution
• Use Δmag of Galactic source to estimate rarity
• Will develop Solar System model for similar approach
Users Committee, NOAO, May 2015 (D2) 14
−1 0 1 2 3
110
010
000
1000
000
1000
0000
0
l96b−60.dat
∆mag
Freq
uenc
y
Min. −1.103 Max. 2.106
0.05% 99.95%−0.080 0.0830.005% 99.995%−0.177 0.3370.0005% 99.9995%−0.548 0.7720.00005% 99.99995%−1.105 1.027
ANTARES: Next Steps
• Run more complex test alerts through system this 2015 • Fully populate annotation databases • Functional prototype in 2016 • Address scaling through 2016 • Run on live alert streams in 2017 • Next proposal will be to make ANTARES generic to
handle the full breadth of user interests • During LSST operations, will likely include NCSA
collaboration • Visualization of ANTARES processes
Users Committee, NOAO, May 2015 (D2) 15
ANTARES: Demo
Users Committee, NOAO, May 2015 (D2) 16
LSST
Initial Ingest
ExistingCatalogs
1,000 images / night
Thumbnails Updated
CalibrationDatadaily
Camera AlertsUp to 10,000 alerts / image
quarterly
Measurement Recalibration
AstroObject & Alert
Association
AstroObject-Annotated Alerts
Feature Evaluation
Extended Alerts
Alert Characterization
Characterized Alerts
Initial Filtering
Level I Alerts
Advanced Feature Evaluation, Characterization and Filtering
Level II Alerts
1,000 / image
100 / image
1.0 / image
Taxonomy of Characterizations
(curated)
filteredalerts
filteredalerts
FilteringTrees
(curated)
CharacterizationAlgorithms(curated)
Locus-AggregatedAnnotated Alerts
PeriodicQueries
DeriveFeatures
New AstroObjectIdentification
PositionEstimation
MovingObjects(curated)
MovingObject
Identification
Feature Algorithms(curated)
LocusAggregation
ANTARES
Theoretical Predictions
AlgorithmVetting
Professionals,citizen scientists
forking
forking
start at imageacquisition
Other External Alerts
External Alert Ingestion
Theorists
Touchstone(curated)
Alert Throttling
Thumbnail Separation
Combo Assembly
Novel Alerts0.1 / image
"Rarest of the Rare"
Annotated Coalesced
Alerts
filtered andnovel alerts
External Alert
Brokers
Researchersand public
Ultra Characterization & Filtering
30 Dec 2014
Alert Replica Coalescing
Thumbnail Reattachment
black = usgreen = not usblue = catalog by-productsred = alert flow
Annotated Coalesced Alert
Rarest Alerts
AggregatedAstroObject Catalogs
(curated)Aggregated
AstroObject Catalogs(curated)Aggregated
AstroObject Catalogs(curated)
forking
Combo ReplicasSelected Alert Replicas
Combo Feature EvaluationAdvanced Feature Evaluation
Combo CoalescingAssociater Load Balancer
Alert Worker 1
Controller
alerts
Antares Load Balancing Diagram
alert status start or send command
send worker
status
send com
mand
Locus Aggregated
DB
LSST Throttler
Stages Configuration
(YAML)
Cached Status Table
for Alerts
maintain
Conductor(person)
Alert Worker Node
Master Node
Alert Worker n
initialize worker
ready(ask Controller)
load new or existing alertrunning
a stage
flush alert data to DB
abort alert worker
complete stage (including divert or fork)
switch to alertresumealert
ready for resume or switch to a different alert
Alert Worker Status Transition Chartstart worker
start alert
camera alerts
provenancemaintenance
Algorithms
Puppet
Github
Deploy codes for worker nodes
Night (fast) Daytime (slow)
reference
machine configuration
stage completed
Camera Alert
Constructor
AstroObject DB
commit
Creating replica or combo and loading data
Background Slides
Users Committee, NOAO, May 2015 (D2) 17
ANTARES: Personnel
• Tom Matheson (NOAO) • Abhijit Saha (NOAO) • Richard Snodgrass (UA Computer Science; PI of NSF
proposal; database expert) • John Kececioglu (UA Computer Science; algorithm expert) • Gautham Narayan (NOAO/UA Postdoc) • Three UA CS grad students • Volunteers
– Tim Axelrod (UA Astronomy) – Rob Seaman (NOAO) – Tim Jenness (LSST)
Users Committee, NOAO, May 2015 (D2) 18
ANTARES: Extant Broker Examples
Users Committee, NOAO, May 2015 (D2) 19
ANTARES: ER Diagram
Users Committee, NOAO, May 2015 (D2) 20
ANTARES: API
Users Committee, NOAO, May 2015 (D2) 21