5-1 dms data products nov. 6, 2013 s&oc data management system design review 3
TRANSCRIPT
5-2
Presentation Outline
Data Definition Review
NIRSpec MSA Metadata File
Imaging Associations
Future Data Product Design
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-3
Data Product References
DMS Requirements Document (JWST-STScI-002249, JWST-RQMT-002941)
Generation of Level 1a Science Data Products (JWST-STScI-002566)
DMS Level 1 & 2 Data Product Design (JWST-STScI-002111)
S&OC Subsystem IRCD (JWST-IRCD-013128, JWST-STScI-000905) Contains specification for data exchanged between S&OC
subsystems
PRDS to Users IRCD Volume III: S&OC Subsystems (JWST-STScI-000949) Contains specification for S&OC data contained in PRD
Definition of the Flexible Image Transport System (FITS), version 3.0, A&A 524, A42 (2010) FITS standard
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-4
Data Level Definition
Science Data Archive
Object Catalog
Level 0: Science telemetry packet files
Level 1a: Original FITS files
Level 1b: Uncalibrated FITS files
Level 2a: Ramp calibrated exposure FITS files
Level 2b: Fully calibrated exposure FITS files
Level 4: Products of data analysis
Extractedobjects
Proposer
Level 3: Calibrated, associated FITS files
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-5
Exposure File Naming Convention
At data levels 1 and 2, the science data files are FITS format files that contain the pixel values for a single exposure from a single detector.
Goals for file name: DMS data user is able to map the data file to the observation input through the
Astronomer's Proposal Tool (APT). Operations staff can map every exposure to a construct in the JWST commanding scripts
that was used to make the observation request.
The file name is based on the following exposure hierarchy: Program (PPPPP) Observation (OOO) Visit (VVV) Visit Group (GG) Parallel Sequence id (S, 1 prime, 2-5 parallel) Activity (AA, base 36) Exposure id (EEEEE)
Example: jw<PPPPP><OOO><VVV>_<GGSAA>_<EEEEE>_<detector >_<suffix> .fits jw12345012009_01101_00012_nrcalong_uncal.fits
Visit ID explicitly embedded in exposure file name
Exposure number resets with each new activity The exposure number within the file name does not necessarily monotonically increase for each
exposure within a single visit.Nov. 6, 2013 S&OC Data Management System Design Review 3
5-7
Relevant DMS Requirements
Nov. 6, 2013 S&OC Data Management System Design Review 3
Req. # Requirement SOC parent
DMS-234 3.3.2.1 Science Data Product GenerationThe DMS shall generate Science Data Products from Science Data.
3.4.5.1.3 Science Data Product Generation [SOC-529]
DMS-133 5.1.6.1 Include PPS Information in Header Keyword ValuesSDP shall populate the FITS data headers with proposal, planning, and scheduling information received from PPS.
3.4.2.1.2 Level 1 Processing [SOC-1689]
DMS-192 6.2.1.1 Generate Science Data ProductsThe DMS shall generate Calibrated Science Data Products from Science Data for all supported modes of the JWST science instruments, as defined in section 3.1.3 of the S&OC Requirements Document (JWST-RQMT-002032, JWST-STScI-000046) under Science Instrument Capabilities.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-460 6.2.1.4 Create Level-2 Data ProductsThe Calibration Software shall create Level 2 Science Data Products as specified in the User Handbooks, OP-07.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-461 6.2.1.5 Create Level-3 Data ProductsThe Calibration Software shall create Level 3 Science Data Products as specified in the User Handbooks, OP-07.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-480 6.2.2.17 Spectra CombinationThe Calibration software shall have the capability of combining spectra extracted from different exposures.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-481 6.2.2.18 Background SubtractionThe Calibration software shall have the capability of subtracting the background from a spectrum.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-482 6.2.2.19 Spectroscopic ExtractionThe Calibration software shall have the capability of extracting a 1-d spectrum from 2-d spectroscopic data.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-549 6.3.1.1 Association DefinitionThe DMS shall define Associations that consist of exposures that may be combined during calibration.
3.4.3.1.2 Science Data Calibration [SOC-513]
DMS-553 6.3.1.5 Contemporaneous Calibration AssociationsThe DMS shall associate contemporaneous calibration exposures with the relevant science exposures.
3.4.3.1.2 Science Data Calibration [SOC-513]
5-8
NIRSpec MSA Reference Documentation
“NIRSpec Data Associations Study: Associating MSA Spectral Data in the Calibration Pipeline and MSA Archive File Structures,” T Beck, J Valenti, J Muzerolle, T Keyes, JWST-STScI-003366, May 24, 2013 Association product study led to definition of data products
“Further Definition of the NIRSpec Science Templates,” D. Karakla, T. Beck, W. Blair, C. Keyes, J. Muzerolle, K. Pontoppidan, D. Soderblom, J. Tumlinson, and J. Valenti, JWST-STScI-002288, 18 March 2011.
“STScI NIRSpec Calibration Pipeline Processing Description – Rev. A,” Tracy Beck, JWST-STScI-001859, 15 June 2011.
NIRSpec MSA Planning Tool “Optimization Strategies for the NIRSpec MSA Planning Tool,” James
Muzerolle, JWST-STScI-001851, 9 December 2009. “MSA Planning Tool – A First Algorithm to Plan an Observation,” Diane
Karakla and the MSA Tool Algorithm Working Group, JWST-STScI-003003, 22 October 2012.
“MSA Planning Tool Design Requirements,” Diane Karakla and the MSA Tool team, JWST-STScI-002971, 4 December 2012.
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-9
NIRSpec MSA Design Summary
Data files Level 1 and 2 science data files based on exposure/detector Level 3 science data files organized by sources DMS metadata file captures MSA metadata from PPS database
Archive paradigm change Save only least combined and most combined data for a program Least combined data – individual exposure data products at level 1
and 2
- No pixel resampling
Most combined data – level 3 data with the most sources
- Changes over time as more visits execute
- New combined products should always be generated using the exposure-level, least combined products as the inputs
- New combined products replace previous versions in the archive
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-10
MSA Data Flow
Nov. 6, 2013 S&OC Data Management System Design Review 3
Data Processing
Proposal Input
MSA Pre-
image
MSA Planning
Tool
PPS Database
APT
Science Data
Processing
Recorded Science Data File
Level 1b Science Data File
Level 2 Calibration
Level 2b Science Data File
Level 3 Calibration
Level 3 Science Data File
MSA Metadata
File
JWSTDetector/
exposure based
Source based
PI
Source Catalog
5-11S&OC Data Management System Design Review 3
NIRSpec MSA Glossary (1/2)
Pre-image – high quality image provided by PI in FITS format used to
select sources for MSA observations relative accuracy of target coordinates from the pre-image must be
known to within 5 mas
Target – position to point the telescope (V2,V3) with fixed orientation
Reference stars – set of 8-20 reference stars to provide relative
astrometry for target acquisition
Catalog – set of sources used to plan observation Not all sources may be observed
Sources – objects of observation in MSA shutter
Target set – set of objects observed simultaneously Target set needs to prevent overlapping spectra for any given
configuration
Nov. 6, 2013
5-12S&OC Data Management System Design Review 3
NIRSpec MSA Glossary (2/2)
Slitlet – set of related micro-shutters For example, one shutter contains the source, other
shutters in the same column contain local background
For medium and high resolution gratings, only one open shutter per row
- Prisms will not spread spectra as far as medium and
high resolution gratings. In that case, multiple
shutters can be open per row without overlapping
spectra.
Rows with a failed open shutter are blocked from use
Rows with a failed closed shutter are tolerated as long as the failed closed shutter is not part of the slitlet
Sources can be dithered within the slitlet by small angle maneuvers- Improves accuracy of background subtraction
- Dithers the source spectrum on the detector between
sub-exposures so that bad detector pixels can be
corrected
Nod – small dither to move source position within
slitlet without MSA reconfigurationNov. 6, 2013
5-13S&OC Data Management System Design Review 3
NIRSpec MSA – DMS Metadata File
Topics to be covered: How DMS/SDP extracts MSA metadata from PPS database Additional MSA information captured in calibration reference files DMS MSA metadata data product design
DMS MSA Metadata File Science Data Processing component extracts information from PPS
and generates the MSA Metadata File MSA Metadata File used by archive users and calibration software
- Archive users do not have access to PPS database
- Provides interface to PPS database for calibration software
Nov. 6, 2013
5-14S&OC Data Management System Design Review 3
NIRSpec MSA Metadata Information Flow
Nov. 6, 2013
SDP
PPS Database
NIRSpec MSA Metadata Product
Calibration
MSA Planning Tool
APT
CRDS
Stuck Open/Closed Shutter
Reference File
MSA to Detector Distortion
Reference File
NIRSpec MSA
configuration info
NIRSpec MSA
PI input
Observ-ation
parameters
Archive
Archive User
5-15S&OC Data Management System Design Review 3
DMS Relevant PPS Tables
Nov. 6, 2013
Observatory Status File:
programobservationvisitpointing_sequence_number
PPS database
nirspec_msa_at_pointing
programobservationvisitpointing_sequence_number
programmsa_config_id
msa_default_state
programmsa_metadata_id
nirspec_msa_config
programmsa_config_id
shutter_quadrantshutter_rowshutter_column
nirspec_target_acq
programmsa_config_id
observationvisitorder_number
reference_star_xreference_star_yreference_star_flux
programsource_id
nirspec_sources
programsource_id
aliascatalog_idradecpre_image_id
nirspec_msa_metadata
programmsa_metadata_id
programsource_id
slitlet_idshutter_quadrantshutter_rowshutter_columnshutter_statebackgroundestimated_source_in_shutter_xestimated_source_in_shutter_y
5-16S&OC Data Management System Design Review 3
Comments on PPSDB Tables
nirspec_msa_at_pointing Allows link between Observatory Status File and NIRSpec MSA PPS database tables msa_default_state = ‘open’ or ‘closed’
nirspec_msa_config Lists shutters not in default state Configuration id unique within a program
nirspec_msa_metadata DMS specific information msa_metadata_id unique within a program Information on shutter level slitlet_id can apply to multiple shutters slitlet_id unique within program shutter_state =‘open’ or ‘closed’: Can force shutters closed to mask bright source background = T or F background shutter can be assigned to multiple sources
PPS to compute source PSF position in shutter in x and y directions of NIRSpec ideal coordinate frame in units of fraction of
shutter length and width Link to nirspec_sources for source information
nirspec_target_acq New visit and target acq required for configuration change Program, observation, visit, and order_number can be used to link to exposure specification 8-20 reference stars per target acq. Link to nirspec_sources for source information
nirspec_sources pre_image_id unique within a program Map pointing to pre-image
Nov. 6, 2013
5-17S&OC Data Management System Design Review 3
PPS nirspec_msa_metadata Table Details
description field data type
values
Metadata id msa_metadata_id I4 unique within a program
Row for the open shutter shutter_row I4 1..365
Column for the shutter shutter_column I4 1..171
Quadrant of the shutter shutter_quadrant I2 1..4
Shutter is opened or closed shutter_state VC(6) Open/Closed
Source associated with this shutter source_id I4 references a source
Is this background background char(1) Y/N
Identifier to indicate the logical group of shutters to which this shutter belongs.
slitlet_id I4 unique within a program
Source PSF position as offset from the shutter center in NIRSpec ideal coordinate frame x direction
estimated_ source_in _ shutter_x
R8 fraction of shutter size in x
Source PSF position as offset from the shutter center in NIRSpec ideal coordinate frame y direction
estimated_ source_in _ shutter_y
R8 fraction of shutter size in y
Nov. 6, 2013
5-18S&OC Data Management System Design Review 3
PPS nirspec_sources Table
Nov. 6, 2013
Field name Type Description Units/range
program I4 Program number 1…99999
source_id I4 Unique ID of a MSA source within a program
alias VC31 Optional name that the observer may give to a source
catalog_id VC20 PI catalog from which the source was taken
ra R8 Coordinates of source in right ascension
degrees
dec R8 Coordinates of source in declination degrees
pre_image_id VC20 Name of the pre-image that was used to measure position for this source
5-19S&OC Data Management System Design Review 3
Additional Files Input to Calibration
Shutter stuck open/closed file(s) and MSA to detector map to be delivered to CRDS through NIRSpec Instrument Team as reference files Information not provided in MSA Metadata File
File(s) to indicate shutters stuck open or closed Stuck shutters may change between time of phase II proposal prep and
visit execution Also provides contrast – fraction of light to get through shutter (even
when shutter open or closed)
Distortion maps Needed for distortion correction in MSA plane
- Not a pixel map, polynomials will be used to characterize distortion
- Distortion in detector plane characterized in SIAF
Used for on-board target acquisition Needed for DMS calibration Same at any given point in time regardless of aperture The mapping file could change from the time of planning to the time of
execution
Nov. 6, 2013
5-20S&OC Data Management System Design Review 3
Science Exposure Header Keyword Information
Pointer to pre-image file name One pre-image per configuration A pre-image can generate multiple configurations
Pointer to stuck open/closed CRDS reference file
Pointer to MSA to detector map CRDS reference file
Pointer to MSA metadata file
Information to map exposure to configuration and metadata Program Observation Visit Pointing sequence number MSA configuration id MSA metadata id MSA default state
Engineering parameter needed to provide grating wheel positionNov. 6, 2013
5-21S&OC Data Management System Design Review 3
Exposure to ID Correspondence
Each exposure has a single configuration id and a single metadata id
Level 1 and 2 files for detectors NRS1 and NRS2 use same configuration id and metadata id
A configuration id can be applied to multiple exposures For example, exposures through multiple gratings with same
configuration
There can be multiple metadata ids within configuration For example, background shutter changes between nods
A metadata id can be applied to multiple exposures, but will typically be applied to a single pointing For example, exposures through multiple gratings
Nov. 6, 2013
5-22S&OC Data Management System Design Review 3
NIRSpec MSA Metadata Data Product
Generated by SDP (Level 1 processing)
Includes information from planning database
FITS format file
One file for each unique MSA configuration Uniqueness defined by program, msa_config_id File name format: jw<visit_id>_<msa_config_id>_msa.fits
- Visit_id 11 characters: <ppppp><ooo><vvv>
- Msa_config_id: 2 base-36 characters
File needs to be associated with all exposures and higher level data products using that configuration Multiple metadata files per data product
Metadata data product generated at time of first exposure using that configuration
Nov. 6, 2013
5-23S&OC Data Management System Design Review 3
FITS File Structure
Primary Header
Source Info Extension
Extension Header
Extension Data Table
Shutter State Extension
Extension Header
Extension DataShutter Info Extension
Extension Header
Extension Data Table
Nov. 6, 2013
5-24S&OC Data Management System Design Review 3
Shutter State Data
Data array to show anticipated state of all shutters Planned shutter state based on PPS database information Calibration can update shutter state information from reference files
and confirmation images
FITS data array values Closed - 0, Open - slitlet_id If shutter open without being assigned a slitlet_id, set data array
value = -1. Imaging mode settings TBD
Nov. 6, 2013
5-25S&OC Data Management System Design Review 3
Shutter Info Extension
Table listing all shutters included in a slitlet Fields would include:
Mimics PPS nirspec_msa_metadata table. Assumes all useful shutters assigned a slitlet_id Background shutters or slitlets can be assigned to multiple sources
- Quadrant, row, and column do not necessarily define a unique table row
metadata_id
shutter_ quadrant
shutter_ row
shutter_column
slitlet_ id
shutter_ state
back ground
source_ id
estimated_ source_in _ shutter_x
estimated_ source_in_ shutter_y
Nov. 6, 2013
5-26S&OC Data Management System Design Review 3
Source Info Extension
Table listing all sources Fields would include:
May include additional information from nirspec_sources_flux
Source id <ppppp>_<nnn> Where nnn is source number within program
Source name Source names standardized Format: <ppppp>_<nnn>_<ra>_<dec> Source names should be assigned within PPS Need to propagate source name from level 2 extensions to level 3 data
files
RA and Dec ICRS reference frame and J2000.0 epoch
- This information to go in the source extension header keywords.Nov. 6, 2013
source_id source name alias catalog_id ra dec
5-28
DMS SDR3 Association Presentations
What is an association (Swade)
How to form an association pool (Swam/Alexov)
How to generate associations (Bushouse)
Separate association design review in 2014 with more details
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-31
Goals of Design
Identify the associated (level 3) data products for each JWST science instrument Develop a naming convention for the associated data products
Identify individual files that will be associated with the data products
Identify the parameters necessary to generate each association
Identify the source of the parameter values
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-32
Association Design Process
Started with designing associations for imaging data products Imaging associations due in DMS Build 3
Considered SODRM programs as use cases Built MIRI and NIRCam imaging association pool and considered
data products for SODRM 93060 - Galactic Stellar Clusters
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-34
Associated Data Products
Imaging Mosaic combined image NIRCam mosaic contains multiple detectors Dither combined image (if no mosaic)
Spectra NIRSpec Extracted 1-d and 2-d spectra over both detectors NIRSpec MSA Source Catalog (level 4) MIRI Extracted 1-d and 2-d spectra MIRI Extracted 1-d and 2-d spectra over both IFU detectors
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-35
Associated Files
Individual Exposures (single detector; levels 1a, 1b, 2a, 2b)
Jitter data
NIRCam Coronagraphic target acquisition
NIRSpec Target acquisition images
NIRSpec Confirmation images
NIRSpec MSA Pre-image
NIRSpec MSA Metadata file
NIRSpec Reference star file
NIRSpec Background exposures (may be shared among exposures)
MIRI Coronagraphic target acquisition images
MIRI Spectroscopic target acquisition images
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-36
Contemporaneous Reference Files
NIRCam flats
NIRSpec Auto-flats
NIRSpec Wave-cals
MIRI Imager internal flats
MIRI MRS internal flats
MIRI Chopped background image MIRI Different visit, no interrupts
Persistence darks (in the distant future)
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-38
Template Definition
Template – APT input used to define individual Observations. An Observation Template is completed in APT which provides the
necessary information for the internal scheduling software to generate the observing sequence (integrations, exposures, and visits) to execute on the telescope.
The templates are selected based on an observing strategy (e.g. MIRI imaging, MIRI low resolution spectroscopy) required for observations.
Only one instrument template can be used for any Observation.
An Observation is a single instance of a template. One-one correspondence exists between an observation and
template instance
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-39
Observation Data Products
In general, level 3 data products do not extend beyond one observation.
A mechanism is needed for PSS to let DMS know about exceptions: Multi-observation mosaics Shared contemporaneous calibration exposures Latent image correction (Defer until the distant future)
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-40
Imaging Data Products
Generate most combined/least combined data products Least combined – exposures processed through level 2b Most combined: Mosaic, Dither, etc.
Generate level 3 data products directly from all relevant level 2b exposures Not from any intermediate data products
Separate observation products generated for Each SI configuration (e.g., filter) NIRCam LW and SW channels
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-41
Dither Data Products
Dither data product generated if no mosaic May extend over multiple visits.
No intermediate data products are generated from sub-pixel dithers.
All exposures of the same target using the same filter combined into a single dither data product. All four NIRCam SW detector exposures combined into dither data
product. LW detector exposures combined into separate data product.
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-43
MIRI Imaging Dither Patterns
• The distribution of sampling positions for the Cycling, 12-point Reuleaux, and 5-point Gaussian patterns, shown at medium scale.
• The origin is the aperture center, where the target is specified.
• Sub-pixel sampling may also be included.• Dither pattern size depends on filter.
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-44
Mosaic Data Products
Mosaic data products combine all possible pointings in an observation. A mosaic is typically completely defined within an observation.
Except mosaic can be split into two observations if Not all tiles have guide stars Multiple position angles improve observing efficiency The first observation fails
No intermediate data products generated (e.g., dither data products).
Do not extend mosaics over programs in the pipeline. Standard pipeline does not address higher level data products.
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-45
Mosaic Observation
Courtesy of Jeff Valenti
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-46
Higher Level Data Products
Pipeline Association Baseline: Combine mosaics/dithers for a single observation (Level 3 data
product)
Higher (> Level 3) data products build by associating data outside routine science data processing pipeline Mechanisms and tools to identify exposures to associate and
facilitate combining exposures will help Example: Multiple observations from a single target in a program
Nov. 6, 2013 S&OC Data Management System Design Review 3
5-48
Future Data Product Design Work
Continue to develop association mechanisms and level 3 imaging data products (DMS Build 3) Association design review in 2014
Add keywords derived from observatory ephemeris to level 1b headers (DMS Build 3) Compute differential velocity aberration correction
Develop FGS guide star image/jitter pipeline and data products (DMS Build 4)
Add keywords from DMS engineering database to level 1b headers (DMS Build 4)
Develop level 2 spectral data products (DMS Build 5) Data array scheme for spectral data
Spectral association mechanisms and level 3 data products (DMS Build 6)
Nov. 6, 2013 S&OC Data Management System Design Review 3