december 11, 2006 des dm - mohr the des dm team tanweer alam 1 dora cai 1 joe mohr 1,2 jim annis 3...

December 11, 2006 DES DM - Mohr

The DES DM TeamThe DES DM Team

Tanweer Alam1 Dora Cai1 Joe Mohr1,2

Jim Annis3 Greg Daues1 Choong Ngeow2 Wayne Barkhouse2 Patrick Duda1 Ray Plante1

Cristina Beldica1 Huan Lin3 Douglas Tucker3

1 NCSA 2 UIUC Astronomy 3 Fermilab

Astronomers Grid Computing, Middleware, Portals Database development, maintenance, Archive web portal NVO lead at NCSA

Senior Developer Oversight Group Randy Butler, Mike Freemon, and Jay Alameda

(NCSA)


Architecture OverviewArchitecture OverviewComponents: Pipelines Archive Portals

Development: 30 FTE-yrs total Current status: 13 FTE-yrs to date


Where are we today?Where are we today?Iterative/Spiral DevelopmentIterative/Spiral Development

Oct ‘04-Sep’05: initial design and developmentOct ‘04-Sep’05: initial design and development basic image reduction, cataloguing, catalog and image basic image reduction, cataloguing, catalog and image

archive, etcarchive, etc

Oct ‘05-Jan’06: DC 1 = deployed DES DM system v1 Oct ‘05-Jan’06: DC 1 = deployed DES DM system v1 Used Teragrid to reduce 700GB of simulated raw data Used Teragrid to reduce 700GB of simulated raw data

[Fermilab] into 5TB of images, weight maps, bad pixel [Fermilab] into 5TB of images, weight maps, bad pixel maps, catalogsmaps, catalogs

Catalogued, ingested and calibrated 50M objectsCatalogued, ingested and calibrated 50M objects

Feb’06-Sep’06: refine & develop Feb’06-Sep’06: refine & develop full science processing through coaddition, greater full science processing through coaddition, greater

automation, ingestion from HPC platforms, quality automation, ingestion from HPC platforms, quality assurance, etcassurance, etc

Oct’06-Jan ‘07: DC 2 = deploy DES DM system v2Oct’06-Jan ‘07: DC 2 = deploy DES DM system v2 Use NCSA and SDSC Teragrid platforms to process 500degUse NCSA and SDSC Teragrid platforms to process 500deg 2 2 in in

grizgriz with 4 layers of imaging in each (equiv to 20% of with 4 layers of imaging in each (equiv to 20% of SDSS imaging dataset, 350M objects)SDSS imaging dataset, 350M objects)

Use DES DM system on workstation to reduce Blanco Use DES DM system on workstation to reduce Blanco Cosmology Survey data (Cosmology Survey data (http://cosmology.uiuc.edu/BCShttp://cosmology.uiuc.edu/BCS ) from ) from MOSAIC2 camera MOSAIC2 camera

Evaluate ability to meet DES data quality requirementsEvaluate ability to meet DES data quality requirements

DC1 Astrometry

DC1 Photometry

DC1 Photometry


DES ArchiveDES Archive

Components of the DES ArchiveComponents of the DES Archive Archive nodes: filesystems that can host DES data Archive nodes: filesystems that can host DES data

files files Large number-- no meaningful limitLarge number-- no meaningful limit Distributed-- assumed to be non-localDistributed-- assumed to be non-local

Database: tracks data using metadata describing the Database: tracks data using metadata describing the files and file locationsfiles and file locations

Archive web portal: allows external (NVO) users to Archive web portal: allows external (NVO) users to select and retrieve data from the DES archiveselect and retrieve data from the DES archive

Try it at Try it at https://des.cosmology.uiuc.edu:9093/des/https://des.cosmology.uiuc.edu:9093/des/


Archive Filesystem StructureArchive Filesystem Structure

host:/${root}/Archivehost:/${root}/Archiveraw/raw/

${nite}/ ${nite}/ (des2006105, des20061006, etc)(des2006105, des20061006, etc)src/src/ original data from telescopeoriginal data from telescoperaw/raw/ split and cross-talk corrected datasplit and cross-talk corrected datalog/log/ logs from observing and processinglogs from observing and processing

red/red/${runid}/${runid}/

xml/xml/ location of main OGRE workflowslocation of main OGRE workflowsetc/etc/ location of SExtractor config location of SExtractor config

files, etcfiles, etcbin/bin/ all binaries required for joball binaries required for jobdata/${nite}/data/${nite}/

cal/cal/ biases, flats, illumination biases, flats, illumination correction, etccorrection, etc

raw/raw/ simply a link to appropriate simply a link to appropriate raw dataraw data

log/log/ processing logsprocessing logs${band1}/${band1}/ reduced images and catalogs reduced images and catalogs

for ${band1}for ${band1}${band2}/${band2}/ and so on for each bandand so on for each band……

cal/cal/ calibration data (bad pixel masks, pupil calibration data (bad pixel masks, pupil ghosts)ghosts)

coadd/coadd/ holds co-added data within ${project}, ${tilename},holds co-added data within ${project}, ${tilename},${runid} ${runid}


DES DatabaseDES Database

Image metadata:Image metadata: Many header parameters (including WCS params)Many header parameters (including WCS params)

All image tags that uniquely identify the DES archive All image tags that uniquely identify the DES archive locationlocation

${archive_site} (fnal, mercury, gpfs-wan, bcs, etc)${archive_site} (fnal, mercury, gpfs-wan, bcs, etc) ${imageclass}= (raw, red, coadd, cal)${imageclass}= (raw, red, coadd, cal) ${nite}, ${runid}, ${band}, ${imagename}${nite}, ${runid}, ${band}, ${imagename} ${ccd_number}, ${tilename}, ${imagetype}${ccd_number}, ${tilename}, ${imagetype}

As long as we adopt a fixed archive structure we can As long as we adopt a fixed archive structure we can very efficiently track extremely large datasetsvery efficiently track extremely large datasets

Simulation metadata:Simulation metadata: We could easily extend the DES archive to track We could easily extend the DES archive to track

simulation datasimulation data Need to adopt some logical structure and we could be up Need to adopt some logical structure and we could be up

and running very rapidlyand running very rapidly


Data Access FrameworkData Access Framework

With DC2 we are fielding grid data movement tools that are With DC2 we are fielding grid data movement tools that are integrated with the DES archiveintegrated with the DES archive

ar_copy: copies dataset from one archive node to another ar_copy: copies dataset from one archive node to another ar_verify: file by file comparison of datasets on two archive nodesar_verify: file by file comparison of datasets on two archive nodes ar_remove: deletes dataset from archive nodear_remove: deletes dataset from archive node

These tools update file locations within the DES databaseThese tools update file locations within the DES database

Data selected using file tags:Data selected using file tags: ar_copy -imclass=raw -nite=des20051005 -imagetype=src mercury gpfs-wanar_copy -imclass=raw -nite=des20051005 -imagetype=src mercury gpfs-wan ar_copy -imclass=red -runid=DES20061120_des20061010_01 mercury mssar_copy -imclass=red -runid=DES20061120_des20061010_01 mercury mss

Underlying grid-ftp tools can vary with archive nodeUnderlying grid-ftp tools can vary with archive node Most sites use Trebuchet, data movement tools integrated with the Most sites use Trebuchet, data movement tools integrated with the

Elf/OGRE middleware development project at NCSAElf/OGRE middleware development project at NCSA FNAL uses globus-url-copy, because there’s an incompatibility with FNAL uses globus-url-copy, because there’s an incompatibility with

Trebuchet listingTrebuchet listing Metadata in the DES db encode the grid-ftp technology as well as Metadata in the DES db encode the grid-ftp technology as well as

combinations of buffer sizes, number of parallel streams, etc for moving combinations of buffer sizes, number of parallel streams, etc for moving “large” and “small” files“large” and “small” files

Recent test by Greg Daues achieved 100MB/s for single copy… Recent test by Greg Daues achieved 100MB/s for single copy… Typically we’ve combined 5 or 6 copies in parallel to achieve total Typically we’ve combined 5 or 6 copies in parallel to achieve total data movement off Mercury of about 50MB/sdata movement off Mercury of about 50MB/s


Archive Portal: Archive Portal: https://des.cosmology.uiuc.edu:9093/des/https://des.cosmology.uiuc.edu:9093/des/

You will be redirected to NVO LoginYou will be redirected to NVO Login


Archive Portal: Image QueryArchive Portal: Image Query


DC2 OverviewDC2 Overview Transferred 10 nights of simulated data from FNAL EnstoreTransferred 10 nights of simulated data from FNAL Enstore

Roughly 3000 DECam exposures {500 deg2 in griz 4 layers deep plus 50 Roughly 3000 DECam exposures {500 deg2 in griz 4 layers deep plus 50 flats/biases each night}flats/biases each night}

Currently: Processed 8 of 10 nightsCurrently: Processed 8 of 10 nights Use Convert_Ingest pipeline to split data {crosstalk corr in this Use Convert_Ingest pipeline to split data {crosstalk corr in this

stage}stage} Typically 20 jobs, each running a couple of hoursTypically 20 jobs, each running a couple of hours Raw data are 600GB for each nightRaw data are 600GB for each night

Submit 62 processing jobs for each of these nightsSubmit 62 processing jobs for each of these nights Each night produces 3.4TB, ~35 million catalogued objects for ingestionEach night produces 3.4TB, ~35 million catalogued objects for ingestion Each job takes around 11hrs… 1 CPU-month to reduce a night of dataEach job takes around 11hrs… 1 CPU-month to reduce a night of data Stages: zerocombine, flatcombine, imcorrect, astrometry, remapping, Stages: zerocombine, flatcombine, imcorrect, astrometry, remapping,

cataloguing, fitscombine, ingestioncataloguing, fitscombine, ingestion Currently some jobs fail because of failures in astrometric refinement…Currently some jobs fail because of failures in astrometric refinement…

Ingest objects into the db Ingest objects into the db Move data from processing platforms to storage cluster and mass Move data from processing platforms to storage cluster and mass

storagestorage Then determine photometric solution for each band/nightThen determine photometric solution for each band/night Update zeropoints for all objects/images for that nightUpdate zeropoints for all objects/images for that night Total data production: 4.8TB raw, 27TB reduced, ~240 million objectsTotal data production: 4.8TB raw, 27TB reduced, ~240 million objects

Still to do: complete processing, co-add all data, extract Still to do: complete processing, co-add all data, extract summary statisticssummary statistics


DC2 ChallengesDC2 Challenges Scale of data- almost overwhelming overwhelmingScale of data- almost overwhelming overwhelming

330GB arrive… 3.4TB produced by next day330GB arrive… 3.4TB produced by next day Ingesting 35 million objects is a challenge-- takes 10 Ingesting 35 million objects is a challenge-- takes 10

hours if ingest rate is 1000 objects/shours if ingest rate is 1000 objects/s Exploring sqlldr alternatives-- most come with a priceExploring sqlldr alternatives-- most come with a price

Moving processed data off compute nodes is a challenge- Moving processed data off compute nodes is a challenge- takes about 10 hours if transfer rate is 100MB/stakes about 10 hours if transfer rate is 100MB/s

New data movement tools making this more reliable and New data movement tools making this more reliable and automaticautomatic

Astrometry problems persistAstrometry problems persist With BCS data we find that astrometry errors are bad With BCS data we find that astrometry errors are bad

enough to produce double sources in a few percent of the enough to produce double sources in a few percent of the images== this translates to at least one failure per co-images== this translates to at least one failure per co-added imageadded image

Taking advice of Emmanuel Bertin to run SCAMP on a per Taking advice of Emmanuel Bertin to run SCAMP on a per exposure basis rather than a per image basis-- new exposure basis rather than a per image basis-- new astrometric refinement framework currently being testedastrometric refinement framework currently being tested


DC2 Photometry and AstrometryDC2 Photometry and Astrometry

Nightly spot checks-- no exhaustive testing so Nightly spot checks-- no exhaustive testing so farfar

Astrometry scatter plots look much like DC1Astrometry scatter plots look much like DC1 Photometry scatter plots don’t look as good, but Photometry scatter plots don’t look as good, but we think we have figured out whywe think we have figured out why

Diffraction spikes/halos added to stars in ImSim2Diffraction spikes/halos added to stars in ImSim2 Done in such a way as to augment total stellar fluxDone in such a way as to augment total stellar flux This leads to an offset in our photometry at the few This leads to an offset in our photometry at the few percent levelpercent level

Detailed statistics await further testingDetailed statistics await further testing What is full distribution of astrometric and What is full distribution of astrometric and photometric errors?photometric errors?

How do both depend on seeing, location on the chip, How do both depend on seeing, location on the chip, intrinsic galaxy parameters, etc…intrinsic galaxy parameters, etc…


Coaddition FrameworkCoaddition Framework Three steps to coadditionThree steps to coaddition

Remapping images to std reference frameRemapping images to std reference frame Determining relative flux scale for Determining relative flux scale for

overlapping remapped imagesoverlapping remapped images Combining remapped images (with filtering)Combining remapped images (with filtering)

DES DM enables a simple automated coaddDES DM enables a simple automated coadd Coadd tiling stored as metadata in the dbCoadd tiling stored as metadata in the db db tools: db tools:

find all tiles associated with imagefind all tiles associated with image find all images associated with tilefind all images associated with tile

ExecutionExecution Reduced images immediately remapped (SWarp) Reduced images immediately remapped (SWarp)

to each tile they overlap (and catalogued)to each tile they overlap (and catalogued) Flux scales determined through (1) db Flux scales determined through (1) db

object matching in overlapping images, (2) object matching in overlapping images, (2) photometric calibration and (3) relative photometric calibration and (3) relative throughput of chips 1-62throughput of chips 1-62

Image combine (SWarp) happens Image combine (SWarp) happens en masseen masse using archive to find correct image using archive to find correct image combinationscombinations Co-add Tiling

DECamImagingLayers


BCS Coadd TestsBCS Coadd Tests Test framework by Test framework by

creating 46 coadd tiles creating 46 coadd tiles that draw images from 10 that draw images from 10 different nights different nights

griz, 36’X36’ with 0.26” griz, 36’X36’ with 0.26” pixelspixels

<1hr job on server with <1hr job on server with 14 drive RAID5 disk 14 drive RAID5 disk arrayarray

Issues:Issues: Flux scaling ignoredFlux scaling ignored Combine algorithm = sumCombine algorithm = sum Science quality?Science quality? Some astrometry failures Some astrometry failures

(double sources)(double sources)

z (3 deep)

i (3 deep)

r (2 deep)

g (2 deep)

4’


Weak Lensing FrameworkWeak Lensing Framework[Mike Jarvis, Bhuv Jain, Gary Bernstein, Erin Sheldon][Mike Jarvis, Bhuv Jain, Gary Bernstein, Erin Sheldon]

Science Strategy: Science Strategy: start from complete object lists and measure shear for each start from complete object lists and measure shear for each

object jointly using all available reduced dataobject jointly using all available reduced data

Draft DES DM strategy:Draft DES DM strategy: Measure shapes of all objects on reduced images as part of Measure shapes of all objects on reduced images as part of

standard reduction and cataloguingstandard reduction and cataloguing Use isolated stars to model PSF distortions across the surveyUse isolated stars to model PSF distortions across the survey Catalog on coadded images to create complete object lists Catalog on coadded images to create complete object lists Use archive tools to select all reduced objects (and images) Use archive tools to select all reduced objects (and images)

for joint shear measurements that include PSF correctionsfor joint shear measurements that include PSF corrections

Implementation just in infancyImplementation just in infancy Shape measurements: one more module for pipeline, db schema Shape measurements: one more module for pipeline, db schema

changechange Modeling PSF distortions: computational (not data) challengeModeling PSF distortions: computational (not data) challenge Complete object lists: Coadd catalogs already available in dbComplete object lists: Coadd catalogs already available in db Final shear measurements: a data challengeFinal shear measurements: a data challenge

Apply data parallel approach grouping by sky coordinates (coadd Apply data parallel approach grouping by sky coordinates (coadd tiling)tiling)

december 11, 2006 des dm - mohr the des dm team tanweer alam 1 dora cai 1 joe mohr 1,2 jim annis 3...

Documents