data-intensive science at johns hopkins university institute for data-intensive engineering and...

Data-Intensive Science atJohns Hopkins University

Institute for Data-Intensive Engineering and Science (IDIES)

Johns Hopkins University

Jordan Raddick

Case Study: Astronomy

• In the “old days” astronomers took photos.• New instruments are digital (< 100 GB/night)• Detectors are following Moore’s law.• Data avalanche: double every 2 years

Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

Why astronomy?

• Because “it’s worthless” (Jim Gray)– No privacy restrictions– No intellectual property– No one wants to sell you data

• Because it’s intrinsically interesting

• And because there’s a lot of it

Astronomy Data

• Astronomers have a few Petabytes now• Data doubles every 2 years• Data is public after 2 years• So, 50% of the data is public• But… how do I get at that 50% of the

data?

Science is breaking

• You can…– search 1 MB in 1 second, transfer for < 1¢– search 1 GB in 1 minute, transfer for $1– search 1 TB in 2 days, transfer for $1,000– search 1 PB in 3 years, transfer for

$1,000,000– …and 1 PB is 5,000 disks

• “Data avalanche” in all fields of science– New approach needed

Data Intensive Science Tools

• Databases instead of files (DBMSs)– Data integrity ensured– Optimized data access (DB indices)– Ability to define methods on data– Do the science inside the database!– Bring the analysis to the data, not vice-versa

• Scalable (parallel) data access• High-speed transport protocols

– Move the data rapidly when necessary

• Asynchronous Web services– Web browsers cannot handle data volumes

Gray’s Laws of Data Engineering

• Jim Gray– Scientific computing is revolving around

data – Need scale-out solution for analysis– Take the analysis to the data!– Start with “20 queries” use cases

Virtual Observatory

• Premise: Most data is (or could be) online• So, the Internet is the world’s best telescope:

– It has data on every part of the sky– In every measured spectral band: optical, x-ray, radio..

– As deep as the best instruments (2 years ago).– It is up when you are up.

The observing conditions are always great (no working at night, no clouds no moons no..).

– It’s a smart telescope: links objects and data to literature on them.

Sloan Digital Sky Survey

• A map of the universe• Telescope at Apache

Point Observatory (Sunspot, NM)– 2.5 meter reflector– ~3 degree field of view– Drift scanning

(telescope stationary, sky appears to move)

An Ambitious Survey

A thousand-fold increase in the amount of data!

• Info content > US Library Info content > US Library of Congressof Congress

•Before SDSS, total number Before SDSS, total number of galaxies with measured of galaxies with measured parameters ~ 200kparameters ~ 200k

•With SDSS, we already With SDSS, we already have detailed parameters have detailed parameters for over 200 Million for over 200 Million galaxies!!galaxies!!

SDSS Data Overview

Data Archive Server (DAS)Data Archive Server (DAS)FITS files (raw data)

Images, spectra, corrected frames, atlas images, binned images,

masksOnline form-based access

Rsync and wget file retrieval

Data Archive Server (DAS)Data Archive Server (DAS)FITS files (raw data)

Images, spectra, corrected frames, atlas images, binned images,

masksOnline form-based access

Rsync and wget file retrieval

Catalog Archive Server (CAS)Catalog Archive Server (CAS)Science parameters extracted to

catalogsStuffed into relational DBMS (SQL

Server)Heavily indexed, optimizedOnline access via SkyServer

Several levels of access, query tools

Catalog Archive Server (CAS)Catalog Archive Server (CAS)Science parameters extracted to

catalogsStuffed into relational DBMS (SQL

Server)Heavily indexed, optimizedOnline access via SkyServer

Several levels of access, query tools

SDSSData

Releasewww.sdss.org

das.sdss.org/DRx-cgi-bin/DAS

www.sdss.orgdas.sdss.org/DRx-cgi-bin/DAS

cas.sdss.org

skyserver.sdss.org

cas.sdss.org

skyserver.sdss.org

The CAS Databases

• Processed data is stuffed into a commercial relational DBMS– Microsoft SQL Server 2000

• Allows fast exploration and analysis - Data Mining• Two versions of the sky: Best and Target

– Target is version of sky on which spectroscopic targets were chosen– Best is latest, greatest processing of the data– 2 DBs for each release, e.g. BestDR3 and TargDR3

• Heavily indexed to speed up access – HTM + DB Indices• Short queries can run interactively.• Long queries (> 1 hr) require a custom Batch Query System.

SDSS Pipelines

SkyServer Web Access

• Supports several levels of SQL access– Novice/casual users

• Radial (cone) and Rectangular search – Intermediate/astro users

• Imaging and Spectro Form Query– Expert

• Free-form SQL, Object Crossid (upload RA/dec list)• CasJobs workbench environment (MyDB)

• Visual tools– ImageCutout service

• Finding Chart, Navigate/browse images, image lists– Explore tool

• Detailed info for each image object, including spectrum

• Downloadable interfaces– Emacs, Java tool (sdssQA), sqlcl (command-line)

http://cas.sdss.org/, http://skyserver.sdss.org/

CAS Workload Management

• Query execution time follows power law

• Vast majority of queries under a minute

• Separate short and long queries

• Execute long queries in batch mode

• Short and long queues (short=1min, long=8hrs)• Strictly limit time of query in a queue

• Provide user workspace DB – MyDB– Reduce network traffic from repeat and intermediate results– Allow sharing of query results between user groups

1

10

100

1000

10000

100000

1000000

0 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

1638

4

3276

8

6553

6

2621

4

5242

8

elapsed

cpu

rows

CasJobs and MyDB

• Batch Query Workbench for SDSS CASBatch Query Workbench for SDSS CAS• Queries are queued for batch execution

– Load balancing – queues on multiple servers– Limit of 2 simultaneous queries per server

• Short (1 minute) queue for immediate mode– Query aborted after 1 minute

• MyDB personal database– 1 GB (more on demand) SQL DB for each user– Long queries write to MyDB table by default– User can extract output (download) when ready– Ability to share MyDB tables with other users via group

visibility

SkyServer

• Entire database of the Sloan Digital Sky Survey– Free to anyone– 4 TB of data– 287 million sky

objects– 1.2 million spectra

Why data to the public?

• All SDSS data available to general public– Commitment

to data sharing– People can look

at any star or galaxy

– Inquiry learning known to be effective (learners answer a question themselves, with their own design)

Browse Data

Explore Data

Search for Data

From Access to Learning

• Not enough just to give public access to data

• How are they using it?• Are they learning anything from it?

– Do they understand what they’re doing and why?*

*Understanding = long-term memory, transfer to new situations (How People Learn)

Formal Education: Projects

SkyServer projects

• Three levels– Kids (K-8)– Basic (high school or Astro 101)– Advanced (skilled and motivated students)

• Research challenges– Independent research (open inquiry) with

data

• Activities created by users

SkyServer projects

Use of educational projects

• 2 of top 20 IPs to hit entire site are K-12 districts– Orlando, FL &

Los Angeles

1.E+5

1.E+6

1.E+7

2001/4 2002/4 2003/4 2004/4 2005/4 2006/4

HitspageViews

Traffic by Month

Citizen science: Galaxy Zoo

• Background: galaxies come in different shapes

• Classifying is hard for computers, easy for people

• So get the public to help!

spiral elliptical

Galaxy Zoo

Galaxy Zoo Discovery

•

• But if you mirror galaxies, it’s still 48/52!• Counter-clockwise excess due to perception• Unintentional psychology discovery!

clockwise (48%) counter-clockwise (52%)

Galaxy Zoo Discovery

• “Hanny’s Voorwerp” (“Voorwerp” = object)

• Strange blue thing found by Dutch teacher Hanny van Arkel

• Most likely “light echo” from a long-dead quasar

• Follow-up time on HST• Galaxy Zoo great for finding

the unexpected!

Galaxy Zoo and learning

• Even people who love science often don’t understand peer review, proposals, etc.

• Want to use Galaxy Zoo to show process of science

• As we write science papers, we share results with volunteers

Galaxy Zoo blog

Other Science: Sensors

• Small, self-contained devices to measure soil temperature, etc.

• Deployed around Maryland

• Study carbon cycle, nesting of turtles, etc.

• Wide range of applications

Other science: Turbulence

• Large-scale simulation of isotropic turbulence in fluid– Imagine shaking a box of water

• FORTRAN code• 1024 x 1024 x 1024 mesh• 10,240 timesteps• Every 10th timestep written to

SQL database

Other science: Turbulence

• Database accessible online through web services

• Can call database from FORTRANor C code

• Like having a supercomputer simulation on your laptop!

Graywulf

• Data-intensive computing architecture– 40 worker nodes, 1440 MB/s each– 6 head nodes, 2100 MB/s each– DDR InfiniBand interconnect

• Deployed mid-2008• Architecture for future projects

Graywulf at SC08

Contact information

Contact Information

Jordan Raddick410-516-8889raddick@jhu.edu