dkist data modelhill, et al., 2007) and the european grid of solar observations (egso) (reardon,...

Project Documentation Document SPEC-0122

Revision A

DKIST Data Model

Friedrich Wöger August 2016

Name Date

Released By: Joseph McMullin

Project Manager 15-Nov-2016

DKIST Data Model

SPEC-0122 Revision A Page ii

REVISION SUMMARY:

1. Date: April 27, 2009 Revision: Draft 1 Changes: Initial version

2. Date: July 21, 2009 Revision: Draft 2 Changes: WCS adopted & keywords added, Decker position added

3. Date: October 7, 2010 Revision: Draft 3 Changes: Comments by Chris Berst, Modification to fit terminology in TN-0138, other minor changes.

4. Date: October 13, 2010 Revision: Draft 4 Changes: ‘Disclaimer’, Exposure & ExposureCycle definition adjusted, Figure texts expanded, Modulation State keyword clarified to be one of the keywords derived well after data acquisition, corrected explanatory text for exposures per accumulator keyword, Data Source of Demodulation Table modified, Camera ID field, other minor changes.

5. Date: March 9, 2012 Revision: Draft 5 Changes: Restructuring to declare DKIST required metadata, simplify the DataSet model.

6. Date: April 10, 2012 Revision: Draft 6 Changes: Add Review Process/OCS deliverables separation in Experiment Control Model

7. Date: November 20, 2012 Revision: Draft 7 Changes: Modified content to new terminology, restructuring of the document, modification of keywords names, and addressing other comments of C. Berst

8. Date: December 13, 2012 Revision: Draft 8 Changes: Added ARMANG to ViSP/NIRSP metadata based on SWG input from R. Casini. Accepted changes to have word update Table and Figure numbers properly. Added requirement sources.

9. Date: July 15, 2013 Revision: Draft 9 Changes: Kevin Reardon comments, changes to new OCS terminology, occulter keywords added

10. Date: September 25, 2013 Revision: Draft 10

DKIST Data Model

SPEC-0122 Revision A Page iii

Changes: Added WeatherStation keywords, to make sure coordinate systems can be established. Other minor changes.

11. Date: December 16, 2013 Revision: Draft 11 Changes: Reworked the way FITS keywords are created for DKIST subsystems in tables in 4.3.0122-0060; FITS required, reserved and WCS reserved keywords are kept. Added necessary line to 4.3.0122-0000 to make sure that columns 47-79 of each FITS line are available to implement the concept.

12. Date: January 4, 2016 Revision: Draft 12

Changes: Introduced FIDOConfiguration & IPTasks, changed “Low” update rates from “Instrument Program” to “DataSetParameters” and added “DataSet” as update rate, ‘highlighted’ required instrument required metadata. Removed instrument specific diagrams.

13. Date: August 2016 Revision: A

Changes: based on feedback from instrument partners and DKIST scientists. Need for IPTask information added, require helioprojective cartesian coordinate system for solar data. Initial formal release.

DKIST Data Model

SPEC-0122 Revision A Page iv

Table of Contents

1 INTRODUCTION ................................................................................................... 1

1.1 OVERVIEW .............................................................................................................. 1 1.2 DOCUMENT SCOPE ................................................................................................. 1 1.3 DEFINITIONS ........................................................................................................... 2 1.4 ACRONYMS ............................................................................................................. 2 1.5 RELATED DOCUMENTS ............................................................................................ 2

1.6 SOURCE ................................................................................................................. 3 1.7 VERIFICATION ......................................................................................................... 3 2 REQUIREMENTS ................................................................................................. 5 3 APPENDIX: DATA REDUCTION LEVELS .......................................................... 42

DKIST Data Model

SPEC-0122 Revision A Page 1 of 43

1 INTRODUCTION

1.1 OVERVIEW

The Daniel K. Inouye Solar Telescope (DKIST), formerly known as the Advanced Technology Solar

Telescope (DKIST), will produce data that may be used by solar physics groups throughout the world. A

way to increase the scientific use and thus value of the data gathered at DKIST is to reach out to as many

solar physicists as possible by providing an “inventory” of the observed data that is searchable for a

variety of keywords. For this reason, it is vital to create metadata that describe the actual data in a

reasonably complete way. From these metadata, a searchable “inventory” or catalog may be created.

The metadata that describe the observed data need to be organized coherently, inspiring the creation of a

data model. The data model provides a description of the processing of all information available to the

system into a useful subset. “Useful” in this context means that known information necessary for a solar

scientist to meaningfully calibrate, analyze and interpret the data is available in the metadata. It is obvious

that this information is instrument dependent and may change in the future. Thus, as for scientific data in

general, a data model needs to be flexible and adaptable to new scientific insights. The DKIST Data

Model adopts the NASA data level numbering (see Appendix, Section 3); the NASA definitions are used

in this text unless explicitly stated otherwise.

Mature data models for solar metadata have been created by the Virtual Solar Observatory (VSO) (e.g.

Hill, et al., 2007) and the European Grid of Solar Observations (EGSO) (Reardon, 2003), and possibly

others. The two mentioned data models are primarily committed to the distribution of the data to

interested solar scientists.

This document defines the requirements and organization, through a data model, of the metadata recorded

at the DKIST, and identifies their source and frequency. The processes that are involved include the

combining of metadata from telescope systems such as the OCS, the TCS or the Adaptive Optics System,

as well as the first light instruments VBI, VTF, ViSP and Cryo- and DL-NIRSP which may or may not be

operated in a multi-instrument setup. Further processing of the metadata may be required to achieve full

compatibility with e.g. the data models of the VSO or that of the EGSO. While the importance of such

compatibility is acknowledged, the focus of this document is on a data model for DKIST that will provide

enough information to achieve such compatibility in subsequent processing steps that are likely to happen

at the NSO Data Center.

The major physical parameters of interest for solar science have been described in Section 2 of Reardon

(2003). For DKIST, it is worthwhile to note that all of these parameters are measured using the properties

of the light that originated at our source of interest, the Sun. No in-situ measurements of the solar

atmosphere are planned for DKIST, the energy regime of which is primarily that of light in the visible and

infrared wavelength regime.

1.2 DOCUMENT SCOPE

This document (herein referred to as the Specifications) includes all specifications related to recording

and formatting at DKIST, including, but not limited to, capture of scientifically valuable metadata. The

specifications in this document serve as origin for DKIST sub-system requirements.

This document may refer to other DKIST project documents for further clarification or explanation only.

DKIST Data Model


1.3 DEFINITIONS

This document uses terminology to describe the DKIST Data Model. In this document, they are defined as

follows:

DKIST-internal interface - an interface between two sub-systems internal to the DKIST, e.g.

between the DKIST OCS and the DKIST DHS.

Non-DKIST-internal interface - an interface between an DKIST sub-system and an entity outside

the scope of the DKIST project, e.g. between the DKIST DHS and the DKIST Data Center.

Fully Processed Accumulator (FPA) - system output of the DKIST Camera Software System; a

FPA is the (only) system output published to the DKIST Data Handling System

(for more details, see SPEC-0098, Section 2.3 - Terminology).

1.4 ACRONYMS

Acronym Meaning

ATST Advanced Technology Solar Observatory

VBI Visible Broadband Imager

ViSP Visible Spectro-Polarimeter

VTF Visible Tunable Filter

DKIST Daniel K. Inouye Solar Telescope

DL-NIRSP Diffraction-Limited Near-Infrared Spectro-Polarimeter

Cryo-NIRSP Cryogenic Near-Infrared Spectro-Polarimeter

VSO Virtual Solar Observatory

EGSO European Grid of Solar Observations

QAS Quality Assurance System of the Data Handling System

PP Processing Pipeline of the Data Handling System for automated data processing

DHS Data Handling System

FPA Fully Processed Accumulator Set (see Section 1.3)

1.5 RELATED DOCUMENTS

SPEC-0001, Science Requirements Document (SRD)

Instrument Science Requirements Documents (ISRDs)

SPEC-0005, Software and Controls Requirements (SCR)

SPEC-0012, Glossary and Acronym List

SPEC-0013, Software Concepts Definitions (SCD)

SPEC-0015, Observatory Control System (OCS)

SPEC-0016, Data Handling System (DHS)

SPEC-0036, Operational Concepts Definition (OCD)

SPEC-0088, Camera Systems Software Specification (CSS)

SPEC-0096, Software Requirements Flowdown

SPEC-0098, CSS Functional Interface

DKIST Data Model


Berst, C. (2010), Camera Control System FITS Header Data, DST Camera Control System

Documentation

Berst, C. (2010), CSS and Instrument Meta-data Concepts, ATST TN-0138

Calabretta, M. R. (2002). Representations of celestial coordinates in FITS. Astronomy &

Astrophysics (395), 1077.

Greisen, E. W. (2002). Representations of world coordinates in FITS. Astronomy & Astrophysics

(395), 1061.

Greisen, E. W. et al. (2006). Representations of spectral coordinates in FITS. Astronomy &

Astrophysics (446), 747.

Hill, F., Bogart, R. S., Davey, A. R., Gurman, J. B., Hourcle, J. A., Martens, P. C., et al. (2007).

Science with the Virtual Solar Observatory: Today and Tomorrow. American Geophysical Union.

W. D. Pence, L. Chiappetti, C. G. Page, R. A. Shaw, and E. Stobie (2010). Definition of the Flexible

Image Transport System (FITS), version 3.0. Astronomy & Astrophysics (524), A42

Reardon, K. (2003). Unified Model for Solar Metadata.

RPT-0038. (2009). ATST VBI Instrument Description Document.

RPT-0039. (2009). ATST ViSP Instrument Description Document.

RPT-0040. (2009). ATST VTF Instrument Description Document.

RPT-0041. (2009). ATST NIRSP Instrument Description Document.

Thompson, W. T. (2006). Coordinate Systems for Solar Image Data. Astronomy & Astrophysics

(449), 791-803.

Wampler, S. G. (2009). ATST Observatory Control System Reference Design Study and Analysis.

NASA (1986), Report of the EOS data panel, Earth Observing System, Data and Information System,

Data Panel Report, Vol. IIa., NASA Technical Memorandum 87777, June 1986, 62 pp. Available at

http://hdl.handle.net/2060/19860021622

JPL/CALTEC (2009). PDS Standards Reference (Chapter 6). Available at

http://pds.nasa.gov/tools/standards-reference.shtml

1.6 SOURCE

This document derives requirements from the Science Requirements Document (SRD), Instrument

Science Requirements Documents (ISRD), and the Operational Concepts Definition (OCD). Some

requirements are considered common engineering practice (CEP), since they would be expected for any

observatory control system.

1.7 VERIFICATION

Included in each major numbered specification listed in this document is a requirement verification

method. These verification methods specify the minimum standards of verification required to ensure that

the individual requirements and specifications are met.

Examples of verification methods include:

• Design Review. Verification by design review means that it is shown during an appropriate

design review that the system meets specification by way of its intrinsic design and configuration.

http://hdl.handle.net/2060/19860021622

http://pds.nasa.gov/tools/standards-reference.shtml

DKIST Data Model


• Analysis. Verification by analysis demonstrates that the design meets the specification through

use of performance modeling metrics.

• Test. Verification by test and/or measurement means that it is demonstrated that the as-built

system meets the specification through measurements of its operation performance. Testing is performed

during acceptance testing and/or as part of a pre-ship readiness review.

• Inspection. Verification by inspection means that visual inspection verifies that the specification

has been achieved on the as-built system during preassembly and/or during Site assembly.

DKIST Data Model


2 REQUIREMENTS

4.3.0122-0000 - Use of the FITS Standard

All DKIST data shall be delivered - combined with its metadata - across non DKIST-internal interfaces in

a format compliant with the FITS 3.0 standard.

All custom, DKIST subsystem specific FITS keyword lines shall reserve the character columns 47-79 for

FITS Comments.

Comment: The FITS 3.0 standard is defined in Pence et al. (2010).

Source: Science (SPEC-0001, 2.8, 4.9), Operations (SPEC-0036, Sect.2.3.5)

Verification: Design Review, Inspection

4.3.0122-0010 - Use of the World Coordinate System

All DKIST data and metadata shall be delivered using a coordinate system that complies with the World

Coordinate System standard as defined in Calabretta (2002) and Greisen (2002, 2006).

All DKIST data and metadata shall be delivered with exactly two coordinate systems,

- one relevant for solar data: Helioprojective Cartesian Coordinates

- one relevant for astronomical objects other than the Sun: Equatorial Coordinates

Comment 1: Some DKIST data will be of spectroscopic, or spectro-polarimetric nature. These type of

data can be handled with the FITS keyword CTYPEn. All WCS relevant keywords are contained in Table

1-a through 1-h of this document.

Comment 2: Whenever possible, solar DKIST data will be delivered with a coordinate system described

in Thompson (2006).



4.3.0122-0020 - Use of the Système Internationale d’Unités

All DKIST metadata delivered across a non DKIST-internal interface shall be in SI (base or derived)

units.

Comment: Internally, DKIST sub-systems, like facility instruments, may store metadata in the DKIST

Header Database (SPEC-0016, requirement 4.3-2320) in non-SI units, which is permitted as long as a

representation in SI units is possible later. The SPEC-0016 requirement 4.3-2601 ensures that these data

are available for later reprocessing of the metadata, if needed.



4.3.0122-0030 - Experiment Data

DKIST shall keep record of any information that defines an DKIST Experiment. This information shall

include:

User ID(s)

Proposal ID

Solar Targets

DKIST Data Model


o target type

o (if applicable) target coordinates (e.g. ‘N50 E30’, ‘north pole’, ‘east limb’)

o (if applicable) target sampling (e.g. mosaicking)

o (if applicable) outside coordination (location, start time, duration)

Instruments

o Instrument ID(s)

o (if applicable) instrument coordination requirements

Setup Requirements

o sufficient resolution values (spatial, spectral, temporal, polarimetric)

o sufficient ranges (spatial, spectral, temporal)

o signal-to-noise ratios

o data processing directives

Telescope Requirements

o GOS / PA&C

o Seeing / AO

o Light Level

o required Coudé BS layout

o required DHS Configuration

Other Requirements

This DKIST Experiment defining information shall be delivered to DKIST systems in a format adequate

for import by the DKIST OCS (SPEC-0015).

Comment: All information necessary for successfully addressing a proposal by carrying out the

corresponding experiment must be contained in the proposal delivered to DKIST.

Comment 2: “Other requirements” could include specialized Coudé optical setups, optics, etc.

Source: Operations (SPEC-0036, Sect.4.3)


4.3.0122-0040 - Attachment of Atmospheric Conditions

All DKIST data delivered across a non DKIST-internal interface shall be accompanied by prevailing

conditions of Earth’s atmosphere during their acquisition, whenever these are available to DKIST

systems. This information shall include the records of r0 (Fried’s parameter) monitored by the DKIST

Wavefront Correction System during data acquisition, and the light level measured by DKIST

Subsystems.

Note: Depending on the Coudé configuration, e.g. during Cryo-NIRSP observations, some of these data

might not be available to DKIST systems.

Source: Operations (SPEC-0036, Sect.2.3.5)


4.3.0122-0050 - Attachment of Thumbnail

DKIST data delivered across a non DKIST-internal interface shall be capable of being accompanied by a

thumbnail whenever such a thumbnail is made available through the DKIST DHS Quality Assurance

System (DHS, SPEC-0016).

Comment: The thumbnail, generated e.g. by a detailed display plugin, is for the purpose of giving an

overview over the data, such as target, quality, etc.

DKIST Data Model




4.3.0122-0060 - Minimal Metadata

All DKIST data delivered across a non DKIST-internal interface shall be accompanied by at a minimum

the metadata detailed in the following Tables 1-a through 1-h at the update rate specified therein.

It shall be possible to access specified metadata, as required in Tables 1-a through 1-h, from the DHS

BDT.

DHS BDT stream header metadata shall take precedence over DHS Header DataBase metadata.

Comment: The DHS BDT metadata transport mechanism is necessary if the metadata is required for DHS

Detailed Display or Plugin Processing.

The rest of this page was intentionally left blank.

DKIST Data Model


Table 1-a: FITS required and reserved keywords required and used by DKIST

Very Low update per InstrumentProgram or less

often …if provided by instrument

Low update per DataSet (as defined by

DataSetParameters) …if provided by instrument

Medium update per CoalescedSysOutput …if provided by instrument

High update per SysOutput …if provided by instrument

InstrumentDefined

update rate defined by the needs of the

instrument used

(could be either one of the above)

…if provided by instrument

Data Source BDT required

Keyword Type Comment

FITS N SIMPLE Strg Mandatory keyword.

Camera Y BITPIX Int Mandatory keyword describing the number of bits per pixel in the data. Permitted values are: 8, 16, 32, 64, -32, -64

DataSetParameters Y NAXIS Int

Mandatory keyword. Value depends on keyword position in file, i.e. whether the keyword is located in the primary HDU, or an extension HDU. Generally, the value field shall contain a non-negative integer no greater than 999 representing the number of axes in the associated data array. A value of zero signifies that no data follow the header in the HDU.

Camera Y NAXISn Int

NAXISn keywords. The NAXISn keywords must be present for all values n = 1, ..., NAXIS, in increasing order of n, and for no other values of n. The value field of this indexed keyword shall contain a non-negative integer representing the number of elements along axis n of a data array. A value of zero for any of the NAXISn signifies that no data follow the header in the HDU (however, the random groups structure has NAXIS1 = 0, but will have data following the header if the other NAXISn keywords are non-zero). If NAXIS is equal to 0, there shall not be any NAXISn keywords.

InstProgram N BUNIT Strg

The physical unit of the data values after application of BZERO and BSCALE according to Pence (2010) (physical_value in BUNIT = BZERO + BSCALE× array_value.) Allowable units and prefixes are stated in Pence (2010), Tables 3, 4, and 5 (and anticipated to be typically ADU)

InstProgram N BZERO Flt see Pence (2010), and BUNIT, default BZERO=0

InstProgram N BSCALE Flt see Pence (2010), and BUNIT, default BSCALE=1

InstProgram N BLANK Int see Pence (2010), and BUNIT, used to assign a number to undefined values when BSCALE and BEZERO are utilized

Clock N DATE Strg UTC Data/Time of HDU creation, in the form: YYYY-MM-DDhh:mm:ss[.sss…]

Clock N DATE-OBS Strg UTC Date/Time of beginning of SysOutput. Time format shall follow the prescriptions for the DATE keyword.

DKIST Data Model



often …if provided by instrument





InstrumentDefined


instrument used





Clock N DATE-BGN Strg Start Date/Time of InstrumentProgram

Clock N DATE-END Strg End Date/Time of InstrumentProgram Keyword empty, if value unavailable

Experiment N ORIGIN Strg

The value field shall contain a character string identifying the organization or institution responsible for creating the FITS file. Value: e.g. “National Solar Observatory”

Experiment N TELESCOP Strg

The value field shall contain a character string identifying the telescope used to acquire the data associated with the header. Value: e.g. “Daniel K. Inouye Solar Telescope”

Experiment N OBSERVAT Strg

A character string identifying the physical entity, located in a defined location, which provides the resources necessary for the installation of an instrument. Value: e.g. “Haleakalā High Altitude Observatory Site”

Experiment N NETWORK Strg

Organizational entity of a series of instruments with similar characteristics or goals that operates in some coordinated manner or produces data with some common purpose. Value: e.g “ GONG”, “SOLIS”

Instrument Y INSTRUME Strg

The value field shall contain a character string identifying the instrument used to acquire the data associated with the header. Value: e.g. “VTF”, “DLNIRSP”, …

Instrument Y WAVELNTH Flt The coarse wavelength band, in nm (nanometer), of

observation

ObsProgram N OBSERVER Strg The value field shall contain a character string identifying the DKIST operator who acquired the data

associated with the header.

TCSParameters N OBJECT Strg The value field shall contain a character string giving a name for the observed object. Applicable standard values are TBD.

ObsProgram N COMMENT Strg This keyword may be used to supply any comments regarding the FITS file.

FITS N END Strg This keyword has no associated value. Bytes 9 through 80 shall be filled with ASCII spaces (decimal 32 or hexadecimal 20).

DKIST Data Model


Table 1-b: FITS WCS and other coordinate system relevant keywords required and used by DKIST


often






InstrumentDefined


instrument used



Data Source

BDT required


Instrument & DKIST

WCSAXESa Int Number of axes in the WCS description. This keyword, if present, must precede all WCS keywords except NAXIS in the HDU. The value of WCSAXES may exceed the number of pixel axes for the

HDU. (see also Pence et al. 2010) Possible values for a: “ “ (blank), or “A” through “Z” Default value: “ “ (blank, indicating the primary WCS name; “A” through “Z” indicate an alternate WCS)

Instrument & DKIST

WCSNAMEa Strg Name of the WCS (see Pence et al, 2010) The index a is present for an alternate WCS (see WCSAXESa)

Instrument CRPIXna Flt The value field shall contain a floating point number, identifying the location of a reference point along axis n, in units of the axis index. This value is based upon a counter that runs from 1 to NAXISn with an increment of 1 per pixel. The reference point value need not be that for the center of a pixel nor lie within the actual data array. Use comments to indicate the location of the index point relative to the pixel. DKIST pointing data will be relative to the boresight of the telescope defined by the WFC Context Viewer that will center an image of the GOS pinhole on its detector using a 10 nm wavelength band centered on 525 nm. The same pinhole image will be used by all instruments as reference for determining the pointing of the instrument in relation to the WFC Context Viewer. The index a is present for an alternate WCS (see WCSAXESa)

Instrument CRDATEna Strg Date of last update of CRPIXn and CDELTn value The index a is present for an alternate WCS (see WCSAXESa) The value will get updated upon realignment of the data source with the WFC Context Viewer (see CRPIXna)

Telescope CRVALna Flt The value field shall contain a floating point number, giving the value of the coordinate specified by the CTYPEn keyword at the reference point CRPIXn. DKIST values for this entry will be determined at the wavelength of the WFC Context Viewer, which covers a band of 10 nm centered on 525 nm. The value will not be corrected for differential

refraction of Earth’s atmosphere. The index a is present for an alternate WCS (see WCSAXESa)

Instrument CDELTna Flt Pixel spacing along axis n (Thompson, 2006) The index a is present for an alternate WCS (see WCSAXESa)

DKIST Data Model



often






InstrumentDefined


instrument used



Data Source

BDT required


Instrument CUNITna Strg The unit of the value contained in CDELTn (Thompson, 2006) The index a is present for an alternate WCS (see WCSAXESa)

Instrument CTYPEna Strg A string value labeling each coordinated axis. This keyword shall be used to describe image data axes, see e.g.

Thompson (2006) This keyword shall be used to describe spectro-polarimeteric axes that the data may have; see also e.g. Section 5.2 in Greisen

(2006) The index a is present for an alternate WCS (see WCSAXESa)

Instrument PCi_ja Flt See e.g. Pence et al. (2010) The index a is present for an alternate WCS (see WCSAXESa)

Instrument PVi_ma Flt See e.g. Pence et al. (2010) The index a is present for an alternate WCS (see WCSAXESa)

Instrument PSi_ma Flt See e.g. Pence et al. (2010) The index a is present for an alternate WCS (see WCSAXESa)

Telescope LONPOLEa Flt See Calabretta (2002) and Thompson (2006)

Telescope LATPOLEa Flt See Calabretta (2002) and Thompson (2006)

Telescope TAZIMUTH Flt raw Telescope azimuth angle in [deg]

Telescope TELEVATN Flt raw Telescope elevation angle in [deg]

Telescope TELTRACK Strg Tracking Mode of the Telescope. Possible values: “None, Fixed Solar Rotation Tracking, Differential Rotation Tracking, Scanning Motion (Random, or Grid, or Raster, or Spiral)”

Telescope TTBLANGL Flt Telescope Coude table angle

Telescope TTBLTRCK Strg Coude table tracking mode. Values: (stepped) parallactic, fixed angle on Sun (e.g. N-S), fixed difference angle between coude and tel. azimuth, fixed coude table angle

DKIST Data Model


Table 1-c: Further DKIST-unique FITS header entries required for the purpose of tracking the data and

the associated scripts and parameters that led to its generation

FITS Keyword

Data Source Comment

ID___nnn OCS Value of the nnn-th data tracking related header item

Required entries for ID___nnn:

Very Low update per InstrumentProgram or less often …if provided by instrument

Low update per DataSet (as defined by DataSetParameters) …if provided by instrument



InstrumentDefined update rate defined by the needs of the instrument used

(could be either one of the above) …if provided by instrument

FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79 Type Comment

001 N DKISTFITSHeaderVersion Strg Version of the DKIST FITS Header format

002 N FileID Strg Unique ID of this FITS file

003 N CheckSum Strg Checksum for data integrity verification.

004 N DataSetID Strg Unique ID of the DataSet associated with these data.

005 N DataSetParametersID Strg Unique ID of the DataSetParameters used to configure the InstrumentProgram associated with these data.

006 N DataSetParameterSequenceID Strg

Unique ID of the DataSetParameterSequence used to configure the InstrumentProgram associated with these data.

007 Y InstrumentProgramID Strg Unique ID of the InstrumentProgram associated with these data.

008 N InstrumentScriptID Strg Unique ID identifying the instrument script that was executed to acquire this data.

009 N InstrumentProgramParametersID Strg Unique ID of the parameters used to configure the InstrumentProgram associated with these data.

010 Y ObservingProgramID Strg Unique ID of the original ObservingProgram associated with the generation of these data.

011 N ObservingProgramAsRunID Strg Unique ID of the ‘as run’ ObservingProgram associated with the generation of these data.

012 N ObservingScriptID Strg Unique ID of the Observing Program Script used to acquire this data.

013 N ObservingProgramParametersID Strg Unique ID of the OCS level parameters used to configure the ObservingProgram associated with these data.

014 N TelescopeParametersID Strg Unique ID of the TCS level parameters used to configure the ObservingProgram associated with these data.

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79 Type Comment

015 Y ExperimentID Strg Unique ID of the experiment associated with the generation of these data.

016

N ProposalID Strg Unique ID of the proposal associated with the experiment associated with the generation of these data.

017 N InvestigatorID Strg Unique ID identifying the Principal Investigator associated with the experiment.

DKIST Data Model


Table 1-d: Further DKIST-unique FITS header entries required for the purpose of capturing important

data information and status

FITS Keyword

Data Source Comment

DKISTnnn ObservingProgram/OCS Value of the nnn-th general DKIST related header item

Required entries for DKISTnnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N OCSMode Strg Mode the telescope was operated in: ‘Auto’: Data were acquired as part of a

regular, automatic execution of an Observing Program ‘Manual’: Data were acquired executing

either a part of an or a complete Observing Program manually.

002 N ObservingProgramRunMode Strg ‘Full’: Data were acquired as part of a

regular execution of an Observing Program ‘Test’: Data were acquired as part of a test

execution of (parts of) an Observing Program

003 Y ObservingProgramTask Strg Observing Task addressed by the Observing Program that led to the acquisition of this data: Observation, Dark, Gain, PolCal, Focus, Alignment, etc

004 Y InstrumentProgramTask Strg Instrument Program Task addressed by the Instrument Program that led to the acquisition of this data: Observation, Dark, Gain, PolCal, Focus, Alignment, etc

005 N FIDOConfiguration Strg The DKIST FIDO configuration in the following format: [M9aStatus,CL2,CL2a,CL3,CL3a,CL4]

006 N InstrumentHealthStatus Strg Worst health status of the instrument during data acquisition Good, Ill, Bad

007 N EngineeringDataFlag Bool 0: Data belongs to a regular science

experiment execution. 1: Data is ‘engineering’ data. Scientific value

of data not guaranteed.

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

008 N StartDateOfValidity Strg Start of date range for validity of data, e.g.

calibration data. Equal to NA, if not applicable (e.g. science data)

009 N EndDateOfValidity Strg End of date range for validity of data, e.g.

calibration data. Equal to NA, if not applicable (e.g. science data)

010 N NumberOfRepeats Int Number (> 0) of repeated DataSetParametersequences within an InstrumentProgram. VBI/DL-NIRSP: Number of field scans to acquire for the observation: N=-1 (loop until interrupted), N>=1

011 N RepeatNumber Int Number (0 < RepeatNumber ≤ NumberOfRepeats) describing the current repetition of the DataSetParametersequence within an InstrumentProgram. VBI/DL-NIRSP: RepeatNumber = Current field scan repetition

012 N RepeatCadence Flt Cadence, in seconds, of the DataSetParameterSequences VBI/DL-NIRSP: Cadence of field scans

013 N LightLevel Int Number of entries in the FITS table accompanying the data containing the value of the telescope light level during data acquisition.

DKIST Data Model


Table 1-e: DKIST Camera Subsystem required items

FITS Keyword

Data Source Comment

CAM_nnn Camera Value of the nnn-th DKIST Camera Subsystem related header

item

Required entries for CAM__nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N CameraUniqueID Strg Unique ID of camera used to acquire the data

002 Y [ms] FPAExposureTime Flt Total duration of exposure to photons, in milliseconds, that resulted in the FPAs within the FPA-set contained in this FITS file.

003 N [Hz] FPAFrameRate Flt Average acquisition rate of FPAs, in Hertz

004 N [ms] CamExposureTime Flt Duration of photosensitive sensor exposure to photons per single sensor read-out

005 N [Hz] CamFrameRate Flt Rate of photosensitive sensor read-outs

006 N [px] ChipDimensionX Int Value describing the number photosensitive pixels on the camera sensor, in X direction

007 N [px] ChipDimensionY Int String value describing the number photosensitive pixels on the camera sensor, in Y direction

008 N HardwareBinningX Int Value describing the binning performed in hardware in pixels, in X direction Equals 1 if disabled

009 N HardwareBinningY Int Value describing the binning performed in hardware in pixels, in Y direction Equals 1 if disabled

010 N SoftwareBinningX Int Binning performed by software in pixels, after hardware binning, in pixels, X Equals 1 if disabled

011 N SoftwareBinningY Int Binning performed by software in pixels, after hardware binning, in pixels, Y Equals 1 if disabled

012 Y NumberOfFPAs Int Number of FullyProcessedAccumulators generated with the DataSetParameters

013 Y CurrentFPA Int Number N of the current FPA generated with the DataSetParameters 1 ≤ N ≤ NumberOfFPAs

014 Y AccumulationsPerFPA Int Number of accumulations per FPA

015 Y [px] ROI1OriginX Int Region Of Interest 1 X - Origin coordinate in pixels

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

016 Y [px] ROI1OriginY Int Region Of Interest 1 Y - Origin coordinate in pixels

017 Y [px] ROI1SizeX Int Region Of Interest 1 X - Size in pixels

018 Y [px] ROI1SizeY Int Region Of Interest 1 Y - Size in pixels

019 Y [px] ROI2OriginX Int Region Of Interest 2 X - Origin coordinate in pixels Keyword not present, if not used

020 Y [px] ROI2OriginY Int Region Of Interest 2 Y - Origin coordinate in pixels Keyword not present, if not used

021 Y [px] ROI2SizeX Int Region Of Interest 2 X - Size in pixels Keyword not present, if not used

022 Y [px] ROI2SizeY Int Region Of Interest 2 Y - Size in pixels Keyword not present, if not used









DKIST Data Model


Table 1-f: DKIST Polarization Analysis & Calibration Subsystem required items

FITS Keyword

Data Source Comment

PAC_nnn PAC Value of the nnn-th DKIST Polarization Calibration and

Analysis Subsystem related header item

Required entries for PAC__nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N UpGOSLinPolarizer Strg Value indicating status of Upper GOS Linear Polarizer slide. OutOfBeam, IDOfLinearPolarizerOpticsInBeam

002 N [deg] UpGOSLinPolarizerAngle Flt Current Rotation Angle of the Linear Polarizer Optics Keyword not present, if not used

003 N UpGOSRetarder Strg Value indicating status of Upper GOS Retarder slide. OutOfBeam, IDOfRetarderOpticsInBeam

004 N [deg] UpGOSRetarderAngle Flt Current Rotation Angle of the Retarder Optics Keyword not present, if not used

005 N [C] UpGOSRetarderTemp Flt Current Temperature of the Retarder Optics Keyword not present, if not used

006 N UpGOSLightSource Strg Value indicating status of Upper GOS Light Source slide. OutOfBeam,

IDOfBroadbandLightSourceInBeam

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

007 N LoGOSObject Strg Value indicating status of Lower GOS Slide Object. FieldStop, Dark, Pinhole, LineGrid, NonRedundantArray, InterferometerSphere, Occulter

008 N [arcs] LoGOSFieldStopDia Flt Aperture diameter of the Lower GOS FieldStop Keyword not present, if not used

009 N [arcs] LoGOSPinholeDia Flt Diameter of the Lower GOS Pinhole Keyword not present, if not used

010 N LoGOSPinholeType Strg Type of Lower GOS pinhole Normal, Inverse

Keyword not present, if not used

011 N [mm] LoGOSLineGridDist Flt Distance between lines of the Lower GOS Line Grid Keyword not present, if not used

012 N LoGOSNRAType Strg Type of the Non-RedundantArray Target Type AirForce, MultiPinhole


013 N [arcs] LoGOSOcculterPosAngle Flt Position angle of the Lower GOS Occulter Keyword not present, if not used

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

014 N DKISTPolarizationCalX12ID Strg Reference ID of the DKIST Polarization Calibration Data (Matrix for M1 and M2) valid for this data (PolCalX-matrices must be selected to be adequate for the wavelength range, and within a ‘valid date range’) Keyword not present, if not used



DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

017 N DKISTPolarizationCalXInstID Strg Reference ID of the DKIST Polarization Calibration Data (Matrix for M7 to the instrument) valid for this data (PolCalX-matrices must be selected to be adequate for the wavelength range, and within a ‘valid date range’) Keyword not present, if not used

DKIST Data Model


Table 1-g: DKIST AO Subsystem required items

FITS Keyword

Data Source Comment

AO___nnn WFC Value of the nnn-th DKIST Wavefront Correction Adaptive

Optics Subsystem related header item

Required entries for AO___nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N HOAOFriedParameterVals Int Number of entries in the FITS table included in this file containing the value of Fried's parameter during data acquisition. Keyword not present, if not used

002 N HOAOLockStatus Bool Lock status of HOAO during data acquisition. 0: HOAO was unlocked for some duration of data acquisition 1: HOAO was locked for the complete duration of data acquisition Keyword not present, if not used

003 [arcs] HOAOLockOffPointingX Flt Current Lockpoint offpointing in X of the HOAO WFS in pixels relative to WFC Context Viewer defined telescope boresight

004 [arcs] HOAOLockOffPointingY Flt Current Lockpoint offpointing in Y of the HOAO WFS in pixels relative to WFC Context Viewer defined telescope boresight

005 [arcs] LimbSensorRadialSetPos Flt Radial set position with respect to the limb as seen at the sensing wavelength of the Limb Sensor = 0 indicates occulting at the limb > 0 indicates over-occulting, i.e. the limb is not visible < 0 indicates under-occulting i.e. the limb is visible Keyword not present, if not used

006 [Hz] LimbSensorRate Flt Frequency with which the Limb Sensor position is read out Keyword not present, if not used

DKIST Data Model


Table 1-h: DKIST Weather Station Subsystem required items

FITS Keyword

Data Source Comment

WS___nnn OCS Value of the nnn-th DKIST Weather Station Subsystem

related header item

Required entries for WS___nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N [m/s] WeathWindSpeed Flt DKIST Local Outside Wind Speed

002 N [deg] WeathWindDirection Int DKIST Local Outside Wind Direction North: 0 deg East: 90 deg South: 180 deg West: 270 deg

003 N [C] WeathOutsideTemperature Flt DKIST Local Outside Tempertature

004 N [%] WeathRelativeHumidity Flt DKIST Local Outside Relative Humidity

005 N [C] WeathDewPoint Flt DKIST Local Outside Dew Point

006 N [hPa] WeathBarometricPressure Flt DKIST Local Outside Barometric Pressure

007 N [ppm] WeathSkyBrightness Flt DKIST Local Outside Sky Brightness

DKIST Data Model


Rationale: The metadata to be collected can be divided into three groups. One group will contain those

data that may change with each SysOutput, one which may change for a CoalescedSysOutput (and thus

DataSet), and one which may only change with an InstrumentProgram.

What a SysOuput, CoalescedSysOutput and InstrumentProgram is comprised of is defined by the systems

acquiring the data. Update rate (per SysOutput, CoalescedSysOutput or InstrumentProgram) and Urgency

are also system dependent characteristics, and therefore must be individually adjustable.

Comment:

The metadata described in Table 1 follow the DKIST Experiment, Instrument and DataSet Model laid out

below.

DKIST Experiment Model

The DKIST Experiment Model is concerned with how an experiment, the fundamental entity that will

lead to data generation at DKIST, is established (see also Wampler & Goodrich (2009)).

Condensed Description:

One or more Users may be involved in one or several Proposals. Each Proposal delivered to DKIST

inspires one Experiment. An Experiment is comprised of one or more ObservingPrograms, each

configured by one set of OPParameters and one set of TCSParameters. An ObservingProgram consists

of one or several InstrumentPrograms. When one InstrumentProgram is executed, it is configured by

one set of IParameters, as well as zero or more DataSets related DataSetParameterSequences

containing one or more DataSetParameters. An InstrumentProgram will ultimately create DataSets.

Detailed Description:

One or more Users can submit one or more Proposals for review. Proposals could be, depending on

operation policies, a combination of several submitted and reviewed proposals should these show

sufficient overlap; details are described in the DKIST OCD document (SPEC-0036).

A Proposal contains a Description. Amongst other things, the Description specifies the type of Solar

Targets - how targets are sampled (e.g. mosaic, rotation), and any outside coordination (location, start

time, duration). In addition, the Description specifies the Instruments in the Proposal and coordination

requirements. Furthermore, the Description specifies Setup Requirements for each instrument. These

Setup Requirements include, as appropriate, sufficient resolution values (spatial, spectral, temporal,

polarimetric), and sufficient ranges (spatial, spectral, temporal), as well as the required signal-to-noise

ratios. The Description also contains Telescope requirements which specify the configuration of the

various telescope subsystems necessary for successful execution of the experiment(s) (GOS / PA&C, AO,

Coudé BS).

One Proposal triggers exactly one Experiment. The Experiment contains all information of the Proposal

in a format that is executable by the DKIST, in particular, the Experiment contains a list of all

ObservingPrograms required to complete that Experiment.

ObservingPrograms are created using the Solar Target, Telescope - encoded in TCSParameters, and

Instrument requirements, and are potentially executed over a long time period (such as synoptic

proposals). The suite of ObservingPrograms that comprises an Experiment includes all the science- ,

calibration- and other activities needed to complete the Experiment; these are encoded in

OPParameters. For efficiency reasons, ObservingPrograms can be associated with several Experiments,

allowing use of the same data for several Experiments - in particular calibration data - in case the data are

valid for these Experiments. On the other hand, ObservingPrograms can - and are expected to - be reused

across Experiments.

DKIST Data Model


One ObservingProgram is comprised of one or more InstrumentProgram(s). One set of IPParameters

encode general settings for one InstrumentProgram. Instrument specific activities, encoded in

DataSetParameters are grouped in DSPSequences if needed. Scripts are created using the Setup

Requirements for each of the instruments, if necessary.

Note: DKIST systems, and in particular the DKIST OCS (SPEC-0015), must have access to all

information starting at the Experiment level and below, which includes the scripts and parameters

required for successfully addressing a proposal.

DKIST Instrument Model

The metadata of DKIST is collected using a scatter-gather approach (Wampler & Goodrich, 2009), which

involves the Components of an DKIST instrument. Each instrument consists of a UTCClock and one or

multiple Components, the properties of which may be vital to the data calibration and performance

analysis, and are defined by the instruments. Each Component might be controlled by

MotionController(s) or ThermalController(s). The Instrument Model illustrated here attempts to

Figure 1: DKIST Experiment Control Model

DKIST Data Model


identify relevant Components of an instrument that could deliver metadata necessary for data calibration

and analysis.

DKIST DataSet Model

The DKIST DataSet Model is a brief description of the data produced at the DKIST.

The model here describes how a DataSet is created and what it is made up of. Taking into account the

requirements listed in this sections, this DataSet Model aligns with that of Section 5.4 in Reardon (2003).

Condensed Description:

Any DKIST system that produces scientifically relevant data (e.g. instruments, or the wavefront

correction system) outputs SysOutput. One or more SysOutputs can be, if necessary after inclusion of

metadata necessary for the Quality Assurance System (QAS) displays and/or the Processing Pipeline (PP)

of the DKIST the Data Handling System (DHS, SPEC-0016), coalesced into CoalescedSysOutput. The

combination of one or more CoalescedSysOutputs will create a DataSet, which may be attached with any

metadata that is not necessarily critical for the QAS and/or the PP. One or multiple DataSets can lead to

Figure 2: The DKIST Instrument Model

DKIST Data Model


one or multiple DataProducts, either produced at DKIST on the summit, or - more commonly - at other

NSO facilities.



4.3.0122-0070 - Attachment of Instrument Specific Metadata

The DKIST shall be capable of appending any data delivered across a non DKIST-internal interface with

further metadata specified by the sub-system that generated the data.

Rationale: Systems such as instruments very likely require metadata beyond those specified in Tables 1-a

through 1-h in order to be able to analyze the data.



Comment 1 - General:

Some instrument specific metadata required by e.g. the instruments have to be derived from the data

delivered by the Data Source (part of either Control Model or Component). The derivation is left to the

system delivered software.

The potential definition of SysOutput, CoalescedSysOutput, DataSet and InstrumentProgram cadences

are given for each system separately, based on the scientific considerations. The potential rate of change

for the Telescope Configuration is defined in each system related comment below.

Comment 2 - VBI:

The Visible Broadband Imager (VBI) is a facility instrument at the DKIST. The moving parts planned are

a filterwheel, with which the wavelengths that are to be observed are selected, camera and lens mounts for

sampling the full field of view and re-focusing, respectively.

In addition to the metadata required by DKIST, it is anticipated that the VBI will need metadata stored

along with all data to be calibrated. More detailed information on what metadata are to be saved appears

in the VBI Design Requirements Document, SPEC-0090.

The following table gives an overview of what additional metadata will likely become necessary.

SysOutput: A SysOutput relates to a single FPA (e.g. one filtergram at one wavelength).

Figure 3: The DKIST DataSet Model

DKIST Data Model


CoalescedSysOutput: CoalescedSysOutput is defined as a set of M (M ≥ 1) observed SysOutputs at a

single wavelength. M is a parameter specified, per wavelength, in the DataSetParameters. Thus, given the

definition above for a SysOutput, a CoalescedSysOutput could consist of M filtergrams. For the VBI,

M=1, 10, 80 could be a common configuration.

DataSet: A data set can be defined as a set of CoalescedSysOutputs. The VBI can sample its full field of

view using a camera stage; the particular pattern with N steps to be used in the field sample is, among

others, defined in the DataSetParameters, and the resulting data belong to the same ‘DataSet’.

InstrumentProgram: An InstrumentProgram is defined by one predefined set of DataSetParameters (a

DataSetParametersSequence, see Figure 1), that lets the VBI cycle through a specified number of

filterwheel positions with specified settings e.g for exposure time. Changing a single parameter will

inadvertently create a new InstrumentProgram.

Telescope Configuration: Needs to be stored on a per DataSet level.

FITS Keyword

Data Source Comment

VBI__nnn VBI Value of the nnn-th VBI Subsystem related header item

Entries for VBI__nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N [nm] InterferenceFilterFWHM Flt Full Width at Half Maximum spectral transmission.

002 N FilterWheelPosition Int Filterwheel position.

003 Y SpatialStepPattern Strg Selected pattern with which the FieldOfView is sampled. Keyword not present, if not used

004 Y NumberOfSpatialSteps Int Value indicating the number of images belonging to the scanned FieldOfView. Keyword not present, if not used

005 Y CurrentSpatialStep Int Value indicating the current image position belonging to the scanned FieldOfView. 1 ≤ CurrentSpatialStep ≤ NumberOfSpatialSteps Keyword not present, if not used

006 Y Processed Bool Value indicating whether the image has been processed by algorithms (e.g. reconstruction algorithms etc.) or not.

DKIST Data Model


Comment 3 - VTF:

The Visible Tunable Filter (VTF) is a facility instrument at the DKIST. The moving parts planned are a

filterwheel, with which the coarse wavelengths that are to be observed are selected, as well as the Fabry-

Perot Interferometers, and various stages for focusing and alignment. With its spectro-polarimetric

capabilities, the instrument will require the polarization calibration unit to be taken into account.

It is anticipated that the VTF will need metadata stored along with all data to be calibrated. More detailed

information on what metadata are to be saved is anticipated to appear in the VTF DRD.

Here, only those metadata are described that is connected to the narrowband channel of the VTF. The

broadband channel of the VTF might have metadata that is similar to that of a VBI channel described in

Comment 2.


SysOutput: A SysOutput relates to a single FPA, e.g. a filtergram encoding a single wavelength step at a

single modulation state (the VTF is a dual beam polarimeter that captures the orthogonal states on

separate cameras).

CoalescedSysOutput: CoalescedSysOutput for a tunable filter based instrument is defined as a set of L

observed SysOutputs at M modulation states. Thus, given the definition above for a SysOutput,

CoalescedSysOutput could consist of L filtergrams with M modulation state. For the VTF, L=1,10 and

M=4 could be common configurations.

DataSet: A data set can be defined as a set of CoalescedSysOutputs. The VTF commonly samples a

spectral line using its Fabry-Perot Interferometer; the N sampling steps to be used are, among others,

defined in the DataSetParameters and the resulting data belong to the same ‘DataSet’.

InstrumentProgram: An InstrumentProgram is defined by one predefined sequence of parameters (a

DataSetParametersSequence, see Figure 1) that lets the VTF cycle through a specified number of

filterwheel positions (wavelengths) with a specified number of settings for the FPI voltages and

modulation states, etc. Changing a single parameter will inadvertently create a new InstrumentProgram.

Telescope Configuration: This data needs to be stored on a per DataSet level.

Note: The VTF, because of the fact that it will encode orthogonal polarization states on two different

cameras will - in spectro-polarimetric mode - typically generate at least two DataSets that are not useful

unless combined together during the calibration process.

How this is achieved is in the calibration process not the purview of this document. However, it is the

purview of this document to capture all the keywords to enable this processing.

FITS Keyword

Data Source Comment

VTF__nnn VTF Value of the nnn-th VTF Subsystem related header item

Entries for VTF__nnn:


DKIST Data Model







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Ty

pe

Comment

001 N [nm] FilterFWHM Flt Full Width at Half Maximum spectral transmission at the current prefilter central wavelength.

002 N FilterWheelPosition Int Filter wheel position of the current prefilter

003 N [nm] WavelengthRangeMinimum Flt Minimum wavelength scanned within the current prefilter bandpass

004 N [nm] WavelengthRangeMaximum Flt Maximum wavelength scanned within the current prefilter bandpass

005 Y NumberOfSpectralSteps Int Number of spectral steps performed by the FPI(s)

006 Y CurrentSpectralStep Int Current spectral step in scan (1 ≤ CurrenSpectralStep ≤ NumberOfSpectralSteps)

007 Y [nm] CurrentSpectralStepWaveL Flt Wavelength at current spectral step

008 Y NumberOfImagesPerSpectralStep Int Number of images acquired at one particular spectral step (and modulation configuration)

009-049 Y Reserved for parameters describing the FPI configurations, TBD

Values of parameters describing the Fabry-Perot Etalon configurations (e.g. peak/central wavelengths, voltages, etc)

050-079 Y Reserved for parameters describing the Polarization Modulator configurations, TBD

Values of parameters that describe the modulation state in the SysOutput (t0, rate, state at t0, number of modulation states, number of accumulations per modulation state, modulation scheme, voltage for each discrete state, demodulation table, etc.)

Comment 4 - ViSP:

The Visible Spectro-Polarimeter (ViSP) is a facility instrument at the DKIST. Configurable parts are

filterselector, with which the coarse wavelengths that are to be observed are selected, slits, lenses, mirrors,

and gratings. With its spectro-polarimetric capabilities, the instrument will require the polarization

calibration unit to be taken into account.

It is anticipated that the ViSP will need metadata stored along with the raw data to be calibrated and that

these metadata are similar to other spectro-polarimeters because of the similar nature of the instrument.

More detailed information on what metadata are to be saved is anticipated to appear in the ViSP Design

Requirements Document.


SysOutput: A SysOutput relates a single FPA, i.e. a single spectrum at a single modulation state snapshot

(the ViSP is a dual-beam polarimeter with both beams captured on the same camera).

DKIST Data Model


CoalescedSysOutput: CoalescedSysOutput for this slit-based Spectro-Polarimeter can be defined as a set

of observed SysOutputs at M modulation states. M is a parameter specified in the DataSetParameters.

Thus, given the definition above for a SysOutput, CoalescedSysOutput could consist of M modulation

spectra, and M=8 might be a common configuration.

DataSet: A data set can be defined as a set of CoalescedSysOutputs. The ViSP commonly samples its full

field of view by scanning its slit across its full field of view; the N sampling steps to be used are, among

others, defined in the DataSetParameters, and the resulting data belong to the same ‘DataSet’.

InstrumentProgram: An Instrument is defined by one predefined sequence of parameters (a

DataSetParametersSequence, see Figure 1) that lets the spectrograph cycle through a specified number of

slit positions, etc. Changing a single parameter will inadvertently create a new InstrumentProgram.

Telescope Configuration: Needs to be stored on a per CoalescedSysOutput level.

FITS Keyword

Data Source Comment

VISP_nnn ViSP Value of the nnn-th ViSP Subsystem related header item

Entries for VISP_nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

Y ArmID Int Identifier for ViSP arm that generated the data: 1, 2, 3

N [deg] ArmPosition Flt Angular position of the arm.

N [mm] ArmFocus Flt Location of the focus stage of the arm.

N FilterID Str Unique ID of the filter in use

N [nm] FilterWavelength Flt Central wavelength of filter in use

N [nm] FilterFWHM Flt Full Width at Half Maximum spectral transmission at the current filter central wavelength in use

N PolarimeterMode Str Polarimeter Mode with which these data were acquired: Full Stokes, Stokes-I, Other(?)

N ModulatorID Str Unique identifier of the modulator.

N ModulationType Str The type of motion used by the modulator: “discrete, continuous”

N [] ModulatorT0 Lon Reference time for the Modulator motion. Unit is TBD.

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

N [Hz] ModulationRate Flt Rate at which states are to be acquired, in Hertz.

N ModulationStateNumAtT0 Int Integer number identifier corresponding to the particular state defined by the position of the modulator at its reference time.

N [deg] ModulationStateAngAtT0 Int Modulator angle corresponding to the modulation state at T0.

Y NumberOfStates Int Number of states to be acquired: NumberOfStates ≥ 1

Y CurrentStateNumber Int Number of the current state: 1 ≤ CurrentStateNumber ≤ NumberOfStates

N GratingID Str Unique identifier of the grating in use.

N [1/mm] GratingConstant Flt Grating constant.

N [deg] GratingBlazeAngle Flt Grating blaze angle.

N [deg] GratingAngle Flt Grating angle with respect to incident beam.

N SlitID Str UniqueID of the slit assembly in use

N [μm] SlitWidth Flt Width of the slit in use

N [μm] SlitSteppingSize Flt Single step size of the slit for this observation Keyword not present, if not used

N [μm/s] SlitScanVelocity Flt Slit move velocity for this observation [ViSP intensity mode only] Keyword not present, if not used

Y NumberOfSpatialSteps Int Value indicating the number of images belonging to the scanned FieldOfView. Keyword not present, if not used

Y CurrentSpatialStep Int Value indicating the current image position belonging to the scanned FieldOfView. 1≤CurrentSpatialStep≤NumberOfSpatialSteps Keyword not present, if not used

N [mm] SlitSelectorPosition Flt Position of the slit selector stage.

N [mm] SlitTranslationPosition Flt Position of the slit translation stage.

N [mm] FoldMirrorPosition Flt Position of the fold mirror stage.

N ViSPSoftwareVersion Strg Version of the ViSP Instrument Software

Comment 5 - DL-NIRSP:

The Diffraction Limited Near Infrared Spectro-Polarimeter (DL-NISRP) is a facility instrument at the

DKIST. Configurable parts are filterwheels, with which the wavelengths that are to be observed are

selected, slits, lenses, mirrors, and gratings. With its optional spectro-polarimetric capabilities, the

instrument may require the polarization calibration unit to be taken into account.

DKIST Data Model


It is anticipated that the DL-NIRSP will need metadata stored along with the raw data to be calibrated and

that these metadata are similar because of the similar nature of the instrument. More detailed information

on what metadata are to be saved is anticipated to appear in the DL-NIRSP Design Requirements

Document.


SysOutput: A SysOutput relates to a single FPA, i.e. a single spectrum and field snapshot using all fibers,

and at a single modulation state snapshot (the DL-NIRSP is a dual polarimeter that encodes both beams

on the same camera).

CoalescedSysOutput: CoalescedSysOutput for this fiber-fed Spectro-Polarimeter can be defined as a set

of observed SysOutputs with M modulation states. M is a parameter specified in the

DataSetParametersSequence. Thus, given the definition above for a SysOutput, CoalescedSysOutput

could consist of M modulation snapshots, and M=8 might be a common configuration.

DataSet: A data set can be defined as a set of CoalescedSysOutputs. The DL-NIRSP commonly samples

its full field of view by scanning the 2D field of view onto its fiber array; the N field sampling steps to be

used are, among others, defined in the DataSetParameters, and the resulting data belong to the same

‘DataSet’.



field positions, etc. Changing a single parameter will inadvertently create a new InstrumentProgram.

Telescope Configuration: Needs to be stored on a per SysOutput level.

FITS Keyword

Data Source Comment

DLN__nnn DL-NIRSP Value of the nnn-th DL-NIRSP Subsystem related header item

Entries for DLN__nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

Y ArmID Strg Identifier for DL-NIRSP arm that generated the data: VIS, JBand, HBand



DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

















N NumberOfDataCycles Int The number of data cycles that should occur for each mosaic tile. A data cycle can be a single or multiple modulation states. A modulation cycle is defined in discrete sampling by the collection of state values. In continuous sampling, it is defined as one half-rotation of the modulator.

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

N CoAddMode Str Defines how data were coadded by the detector. Values are defined as follows: State: All coadded exposures for a single

state are acquired prior to progressing to the next modulation state (discrete modulation). Sequence: The full discrete modulation

sequence is cycled through a number of times equivalent to the number of coadds. Half-rotation: States of the first half

rotation are coadded with comparable states of the second half rotation. None: All exposures are saved without

coadding.

N FeedOpticsFRatio Str Configuration of the DL-NIRSP Feed Optics: F/24, F/62

N IFUInBeam Str Value indicating BiFOIS-IFU that was used for this observation: IFU-36, IFU-72

N SlitMode Str Value indicating whether DL-NIRSP was operated in single- or multi-slit configuration: Single, All

N SlitSelection Int Identifier of the slit used when DL-NIRSP is operated in single slit mode: 1,2,3,4,5


Y SpatialStepPattern Strg Selected pattern with which the FieldOfView is sampled. Keyword not present, if not used

Y DitherMode Bool Boolean indicating whether the DL-NIRSP dithering mode was activated or not. When enabled, a sub-unit field shift is used between successive completed FieldSamples: 0, 1

Y NumberOfSpatialStepsX Int Value indicating the number of images in horizontal direction belonging to the scanned FieldOfView. Keyword not present, if not used

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

N [deg] StepCenterPositionX Flt Angular position of the center position in the horizontal direction of the field scan given in units of the field scanning mirror motorized stage. Default position will be a property updated when the instrument is co-aligned with other instruments using a GOS target.

Y [deg] StepSizeX Flt Angular step size in the horizontal direction given in units of the field scanning mirror motorized stage. Default value will be

calculated according to IFU, slits, and feed optics selection.

Y CurrentSpatialStepX Int Value indicating the current image position in horizontal direction belonging to the scanned FieldOfView. 1 ≤ CurrentSpatialStepX ≤ NumberOfSpatialStepsX Keyword not present, if not used

Y NumberOfSpatialStepsY Int Value indicating the number of images in vertical direction belonging to the scanned FieldOfView. Keyword not present, if not used

N [deg] StepCenterPositionX Flt Angular position of the center position in the vertical direction of the field scan given in units of the field scanning mirror motorized stage. Default position will be a property updated when the instrument is co-aligned with other instruments using a GOS target.

Y [deg] StepSizeY Flt Angular step size in the vertical direction given in units of the field scanning mirror motorized stage. Default value will be calculated according to IFU, slits, and feed optics selection.

Y CurrentSpatialStepY Int Value indicating the current image position in vertical direction belonging to the scanned FieldOfView. 1 ≤ CurrentSpatialStepY ≤ NumberOfSpatialStepsY Keyword not present, if not used

Comment 6 - Cryo-NIRSP:

The Cryogenic Near Infrared Spectro-Polarimeter (Cryo-NISRP) is a facility instruments at the DKIST.

Configurable parts are filterwheels, with which the wavelengths that are to be observed are selected, slits,

DKIST Data Model


lenses, mirrors, and gratings. With its optional spectro-polarimetric capabilities, the instrument may

require the polarization calibration unit to be taken into account.

It is anticipated that the Cryo-NIRSP will need metadata stored along with the raw data to be calibrated

and that these metadata are similar because of the similar nature of the instrument. More detailed

information on what metadata are to be saved is anticipated to appear in the Cryo-NIRSP Design

Requirements Document.


SysOutput: A SysOutput relates a single FPA, i.e. a single spectrum at a single modulation state snapshot

(the Cryo-NIRSP is a dual-beam polarimeter with both beams captured on the same camera).

CoalescedSysOutput: CoalescedSysOutput for this slit-based Spectro-Polarimeter can be defined as a set

of observed SysOutputs at M modulation states. M is a parameter specified in the DataSetParameters.

Thus, given the definition above for a SysOutput, CoalescedSysOutput could consist of M modulation

spectra, and M=8 might be a common configuration.

DataSet: A data set can be defined as a set of CoalescedSysOutputs. The Cryo-NIRSP commonly samples

its full field of view by scanning its slit across its full field of view; the N sampling steps to be used are,

among others, defined in the DataSetParameters, and the resulting data belong to the same ‘DataSet’.



slit positions, etc. Changing a single parameter will inadvertently create a new InstrumentProgram.

Telescope Configuration: Needs to be stored on a per SysOutput level.

FITS Keyword

Data Source Comment

CRSP_nnn Cryo-NIRSP Value of the nnn-th Cryo-NIRSP Subsystem related header

item

Entries for CRSP_nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

N ArmID Int Identifier for Cryo-NIRSP arm that generated the data: Spectro-Polarimeter, Context Viewer


DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment


















N SlitAssemblyID Str UniqueID of the slit assembly in use

N [μm] SlitWidth Flt Width of the slit in use

N [μm] SlitSteppingSize Flt Single step size of the slit for this observation Keyword not present, if not used

N [μm/s] SlitSteppingVelocity Flt Slit move velocity for this observation [intensity mode only] Keyword not present, if not used

Y NumberOfSpatialSteps Int Value indicating the number of images belonging to the scanned FieldOfView. Keyword not present, if not used

Y CurrentSpatialStep Int Value indicating the current image position belonging to the scanned FieldOfView. 1≤CurrentSpatialStep≤NumberOfSpatialSteps Keyword not present, if not used

DKIST Data Model








FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

N SpatialStepPattern Strg Selected pattern (scan direction) with which the FieldOfView is sampled. Keyword not present, if not used

N PickOffMirrorPosition Strg Value indicates the position of the Pick Off Mirror used for this observation

N AttenuationFilterID Strg ID of Attenuation Filter in beam. Keyword not present, if not used

N ContextViewerFilterWheel1 Strg Value indicates the named position active in the Cryo-NIRSP Context Viewer Filterwheel 1

N ContextViewerFilterWheel2 Strg Value indicates the named position active in the Cryo-NIRSP Context Viewer Filterwheel 2

Comment 7 - Wavefront Correction System:

The Wavefront Correction System (WFC) is a fixed support facility at DKIST, and part of the Telescope

Control System (TCS). Its primary purpose is to detect and correct optical aberrations induced by both

telescope alignment errors and atmospherically induced refractive index fluctuations.

Data recorded by the WFC are potentially useful for post-facto algorithms to reduce residual aberrations

in data gathered with DKIST’s scientific instrumentation. WFC will initially (at first light) deliver three

separate data products in three separate, continuous streams:

1. shift values measured by the HOAO wavefront sensor

2. the prevailing commands for the actuators of the correcting deformable mirror

3. the error commands for the actuators of the correcting deformable mirror

It is anticipated that the WFC will need metadata stored along with each raw data product to ensure that

the recorded data is scientifically meaningful.


The context viewer of the WFC Subsystem will have metadata that are very similar to that of a VBI

channel described in Comment 2.

InstrumentProgram: N/A

SysOutput: A SysOutput for WFC relates to an array consisting of either computed shift or actuator or

actuator error values for a given single measurement.

CoalescedSysOutput: CoalescedSysOutput for the WFC output is defined as a set of N observed

SysOutput. N is a parameter that is set by the WFC subsystem to ensure efficient use of the bandwidth of

the DHS BDT.

DKIST Data Model


Telescope Configuration: Needs to be stored on a per CoalescedSysOutput level.

FITS Keyword

Data Source Comment

WFC__nnn WFC Value of the nnn-th WFC Subsystem related header item

Required entries for WFC__nnn:







FITS

Keyword

number

nnn

BDT

required

FITS Comment

in columns 47-79

Type Comment

001 N [nm] FilterFWHM Flt Full Width at Half Maximum spectral transmission at the current prefilter central wavelength.

002 Y HOAOFrameLockStatus Bool Lock status of the WFC HOAO subsystem for the current frame 0: HOAO was unlocked during frame time 1: HOAO was locked during frame time

003 Y HOAOReconstructionMatrixID Strg ID of the WFC HOAO Reconstruction matrix currently in use

004 Y [arcs] HOAOLockOffPointingX Flt Current Lockpoint offpointing in X of the HOAO WFS in pixels relative to WFC Context Viewer defined telescope boresight

005 Y [arcs] HOAOLockOffPointingY Flt Current Lockpoint offpointing in Y of the HOAO WFS in pixels relative to WFC Context Viewer defined telescope boresight

006 Y HOAOCtrlLoopPValue Flt Value of the current proportional value of the WFC HOAO PI control loop

007 Y HOAOCtrlLoopIValue Flt Value of the current integral value of the WFC HOAO PI control loop

008 Y [Hz] HOAOWFSFrameRate Int Frame rate of the WFC HOAO Wavefront Sensor

Furthermore, for the WFC subsystem, the BDT requires scale factors for each stream:

Very Low update per InstrumentProgram or less often

Low update per DataSet (as defined by DataSetParameters)

Medium update per CoalescedSysOutput

High update per SysOutput

DKIST Data Model






WFCSubSys BSCALE Flt Scale factor to achieve conversion of the data into BUNIT, following the FITS standard.

WFCSubSys BZERO Flt Offset value to achieve conversion of the data into BUNIT, following the FITS standard.

WFCSubSys BUNIT Strg The physical unit of the data values after application of BZERO and BSCALE (physical_value in BUNIT = BZERO + BSCALE× array_value.) Possible values:

“radians of wave @ 525nm”

“nm” (nanometer)

DKIST Data Model


3 APPENDIX: DATA REDUCTION LEVELS

Table 2 below summarizes two well-known definitions of data reduction levels; one based on NASA

(1986) and one based on JP/CALTECH (2009) (which itself is based on the National Research Council’s

Committee for Data Management and Computation report, CODMAC). The descriptions include slight

modifications to better match solar ground-based observations:

NASA CODMAC Type Description

N/A Level 1 Raw Data Telemetry data with data embedded.

Level 0 Level 2 Edited Data Corrected for telemetry errors and split or

decommutated into a data set for a given

instrument. Sometimes called Experimental Data

Record. Data are also tagged with time of

acquisition (and potentially additional records).

Level 1A Level 3 Calibrated Data Edited data that are still in units produced by the

instrument, but have been corrected so that

values are expressed in or are proportional to

some physical unit such as radiance. No

resampling, so edited data can be reconstructed.

Level 1B Level 4 Resampled Data Data that have been resampled e.g. in the time or

space domains in such a way that the original

edited data cannot be reconstructed. Could be

calibrated in addition to being resampled.

Level 2

Level 3

Level 4

Level 5 Derived Data Derived results, as maps, reports, graphics, etc.,

corresponding to NASA Levels 2 through 4.

Nasa Level 2: Derived variables (e.g., velocities,

magnetic field strengths, etc.) at the same

resolution and location as the NASA Level 1A/B

source data.

Nasa Level 3: Variables mapped on uniform

space-time grid scales, usually with some

completeness, and consistency properties (e.g.,

missing points interpolated, complete regions

mosaicked together from multiple data sets, etc.).

Nasa Level 4: Model output or results from

analyses of lower-level data (i.e., variables that

were not measured by the instruments but instead

are derived from these measurements).

N/A Level 6 Ancillary Data Nonscience data needed to generate calibrated or

resampled data sets. Consists of instrument

gains, offsets, pointing, etc.

N/A Level 7 Correlative Data Other science data needed to interpret data sets

according to the proposal. May include e.g. co-

temporal space-based data observations.

N/A Level 8 User Description Description of why the data were required, any

peculiarities associated with the data sets, and

enough documentation to allow secondary user

to extract information from the data.

Table 2: Two well-known definitions of data reduction levels.

DKIST Data Model


It is worthwhile noticing that DKIST delivers with some exceptions Level 0 data, attached with

CODMAC Level 7 and 8, and in part CODMAC Level 6 data across non-DKIST internal interfaces.

Calibration data such as gain and offset data for instruments cannot be attached with the Level 0 science

data because of their potentially large volume, even though they belong in the CODMAC Level 6 data

category.