python user group november 2013, sql(alchemy) and transient detection

Transient Detection and SQL

Gijs Molenaar !

Anton Pannekoek Institute (UvA) !

@gijzelaerr gijs@pythonic.nl

LOFAR

• Radio telescope

• ~10–240 MHz

• Software telescope

• omni-directional

AARTFAAC

• Lofar sub project

• 24/7 full sky transient detection

• ‘Changes’

• 99% Python code

Transients

• Super-nova

• Gamma-ray burst

• Pulsars

• exo-planets

• ?

Images

• 10-240 MHz

• Split in 10 bands

• 1 image cube per second

• future 10 images per second

Data

• Source

• Position

• Flux (intensity)

• Shape (2d gaussian)

Data rate

• 500 - 5000 inserts per second

• Higher in other modes

Calculations

• Associate in frequency

• Associate in time

Quite a lot of data

• move calculations to data (base)

nonosql

• Structured data

• Natural sharding by sky position

• MonetDB

Why MonetDB

• Dutch (CWI)

• Fast with huge datasets

• Developers next door

calculations to data

• No moving around data

• one huge big monolithic query per ‘operation’

Example

HORRIBLE• Huge monolithic queries

• hard to read

• hard to reason about

• hard to test

• hard to maintain

• hard to port

Solution?

• Functions?

NO!

• DB revisions, versioning hell

• For every code change new version

• Schema migrations

• Init functions on startup?

• DB specific

SOLUTION

Why?• Object Relational Mapper

• Better Query Builder Language!

• Support multiple database dialect

• Functions per dialect

!

• SQLAlchemy -> SQL is like C -> assembler

example

Why not earlier?

• Started by non programmers

• No MonetDB dialect for SQLAlchemy

sqlalchemy-monetdb

• pip install sqlalchemy-monetdb

• https://github.com/gijzelaerr/sqlalchemy-monetdb

• https://github.com/gijzelaerr/djonet

• (banana)

Questions?