LSST VAO Meeting

March 24, 2011Tucson, AZ

Headquarters SiteHeadquarters FacilityObservatory ManagementScience OperationsEducation and Public Outreach

Archive SiteArchive CenterNightly Reprocessing

Data Release Production Long-term Storage (copy 2)Data Access Center

Data Access and User Services120 - 330 TFLOPS17 - 140 PB Disk5 - 100 PB Tape

Base SiteBase FacilityAlert Production

Long-term storage (copy 1)Data Access Center

Data Access and User Services50 - 60 TFLOPS9 - 110 PB Disk5 - 100PB Tape

Summit SiteSummit Facility

Telescope and CameraData Acquisition

Crosstalk Correction

Data Management System Sites and Centers

Data Management Presentation to LSSTC Board 3

ChileDAC

Tucson

La Serena

Pachon

Santiago SaoPaolo

Miami

NCSA

Data80Gbps

TCS/OCSControl20Gbps

10/100GbpsReuna

10/100GbpsLauren/Clara

10/100Gbps Amlight

10/100GbpsNLR/I2

TeraGrid

(Telefonica&Telmex)

DMS long-haul network design

• Mountain summit – Base is only new fiber, 100 Gbps capacity• Inter-site Long-Haul links on existing fibers

– Chile-US– Archive – HQ– DAC – End Users

• 1 Gbps 2011, 10 Gbps 2015, 100 Gbps 2025• Site End Equipment is available today

Network Design Document-7354

Panama LosAngeles

Data Management Presentation to LSSTC Board 4

DMS cyber-security plan

• Threats– Denial of service

attacks– Break-in attempts– Internal attacks– Code injection– Personal computer

malware, viruses, etc.

• Best practices– Monitor

infrastructure– Partition resources– Grant limited

privileges– Simulate threats

Cyber-security Plan-9733

LSST DM R&D Plan To Date

LSST DM R&D Plan To Be Done

DMS Data Challenge Infrastructure

SPIE 2010 June 30, 2010 San Diego, CA

Data Management Presentation to LSSTC Board 8

DMS Middleware - Pipeline processing• Middleware runs in full production

mode as operational system will do

• Have executed in parallel on hundreds of processing cores

• Performance can be increased by adding more hardware, input/output remains most constraining

• Cross-project workshops conducted with DES, JWST, ODI, UWis Condor group to leverage lessons learned

• Provenance and Fault-Tolerance demonstrations by PDR

DMS Middleware – Database Architecture

• Current relational database technology cannot handle LSST data volumes• Solution: make many

databases look like one to users and applications• Scale by adding database

instances/servers without software changes• Full prototype now

implemented

9Supercomputing 2010

NSF ReviewDecember 15-17, 2009Tucson, AZ 10

Data Challenge Applications ScopeDC3a

DC3a

DC2

DC2

DC3b

DC3bDC4

DC4

DC4

DC1

DC1

Data Management Presentation to LSSTC Board 11

DMS Applications Status• Since the start of R&D, spent on software development and

integration– $8M (out of $10M total R&D spent)– 90 FTE years (funded and in-kind, average 40% on project)

• By comparison– Terapix (unknown, 7 people, started in 1997)– 2MASS (165 FTE years)– SDSS (145 FTE years low estimate, 215 FTE years high estimate)– Above estimates are from respective project managers/technical leads