system design/implementation and support for build 2 pds management council face-to-face mountain...

System Design/Implementation and Support for Build 2

PDS Management Council Face-to-FaceMountain View, CA

Nov 30 - Dec 1, 2011

Sean Hardman

Topics

• Overview

• Key Drivers and Requirements

• Build 2 Deliverables

• Build 2 Deployment

• Next Steps

Nov 30 - Dec 1, 2011 System Design and Support for Build 2 2

Architectural ElementsAddressed in Build 2


Topics

• Overview




• Next Steps


Build 2 Drivers

• Transition the central catalog to the registry infrastructure.- Ingestion of PDS3 and PDS4 catalog-level metadata

moving forward.- Facilitate catalog-level search of the registered

metadata.

• Provide PDS4 software tools to support early adopters.- Includes tools for designing, generating and

validating PDS4 product labels.


Core Concepts

• Service-Based Design- Support remote access to data and services to bring

the federation together both for ingestion and distribution.

• System of Registries- Adopt a system of registries to support improved

tracking and access.

• Common Search- A publicly available layer facilitating search across PDS.

• Enhanced Tool Suite- A tool-based approach is still appropriate for certain

functions.


Level 3 RequirementsPertaining to Build 2

Requirement Implementation Build

1.5.1 PDS will provide tools to assist data producers in generating PDS compliant products

• Design Tool (Oxygen or Eclipse) facilitates XML Schema design.

• Generate Tool facilitates product label generation.• Atmospheres Node tool supports product label

generation.

1

2

1, 2

1.5.2 PDS will provide tools to assist data producers in validating products against PDS standards

• Validate Tool validates product labels against their schemas. Additional functionality planned in future builds.

1, 2, 3

1.5.4 PDS will provide documentation for installing, using, and interfacing with each tool

• All components delivered to date provide documentation for installation and operation.

1, 2, 3


Level 3 RequirementsPertaining to Build 2 (cont)


2.2.2 PDS will track the status of data deliveries from data providers through the PDS to the deep archive

• Harvest Tool registers the data products that make up a data delivery.

• Registry Service accepts and manages data product registrations.

• Report Service manages metrics and facilitates reporting.

1, 2

1, 2

1, 2

2.6.2 PDS will design and implement a catalog system for managing information about the holdings of the PDS

• Harvest Tool registers the data products that make up a data delivery.

• Registry Service accepts and manages data product registrations.

1, 2

1, 2

2.6.3 PDS will integrate the catalog with the system for tracking data throughout the PDS

• All components delivered to date provide APIs for ease of integration.

1, 2, 3




2.8.2 PDS will maintain a distributed catalog system which describes the holdings of the archive

• Registry Service designed for distributed deployment with replication of contents to an aggregate registry instance.

2

2.8.3 PDS will provide standard protocols for locating, moving, and utilizing data, metadata and computing resources across the distributed archive, among PDS nodes, to and from missions, and to and from the deep archive

• All components delivered to date provide APIs for ease of integration and access.

• Search Service specifically specifies a protocol for data discovery.

1, 2, 3

2, 3

2.8.7 The PDS architecture will enable non-PDS developed tools to access PDS holdings and services

• All components delivered to date provide APIs for ease of integration and access.

1, 2, 3




2.9.2 PDS will ensure that online interfaces comply with required NASA Guidelines

• All web-based applications shall be 508 compliant.

2, 3

2.9.3 PDS will meet U.S. federal regulations for the preservation and management of data.

• Report Service secures transfer and access to metrics.

2

2.10.3 PDS will ensure that appropriate mechanisms are in place to prevent unauthorized users from compromising the integrity of PDS systems and data

• Security Service performs this function.

2

3.1.1 PDS will provide online interfaces allowing users to search the archive

• Registry Service provides an interface for harvesting product metadata.

• Search Service provides the capability to index and search the harvested metadata.

1, 2

2, 3


Level 3 RequirementsPertaining to Build 3 and Beyond

• 3.1.2 PDS will provide online interfaces for discipline-specific searching• 3.1.3 PDS will allow products identified within a search to be selected for

retrieval• 3.2.2 PDS will provide a mechanism for offline delivery of portions of the

archive to users• 3.2.3 PDS will provide mechanisms to ensure that data have been

transferred intact• 3.3.2 PDS will provide a capability for opening and inspecting the contents

(e.g. label, objects, groups) of any PDS compliant archival product• 3.3.3 PDS will provide tools for translating archival products between

selected formats• 3.3.4 PDS will provide tools for translating archival products between

selected coordinate systems• 3.3.5 PDS will provide tools for visualizing selected archival products• 4.2.4 PDS will provide a mechanism to upgrade products or data sets

which do not meet usability requirements (e.g., data sets from old missions)


Topics

• Overview




• Next Steps


Build 2 Deliverables

• Initial Distribution Subsystem- This includes the Search component and an updated Data

Search interface at the EN.- Replicate EN functionality with PDS 2010 infrastructure.

• Ingestion Subsystem- Completed core components with support for PDS3 products

and keeping pace with changes in the data model.

• Tools- A stable suite of tools for early adopters including Design,

Generate and Validate tools.

• Operations- Configuration of the Report component, selection of an off-

the-shelf product for Monitor, and modification of existing search interfaces.


Build 2 DeliverablesCore System Components

• Registry Service- Provides functionality for tracking, auditing, locating,

and maintaining artifacts (e.g., products, services, etc.) within the system.

• Harvest Tool- Provides functionality for capturing and registering

product metadata with the Registry Service.

• Search Service- Provides functionality for searching the metadata

contained in the Registry Service instances.


Build 2 DeliverablesSupport System Services

• Security Service- Provides the authentication and authorization

functions for the system.

• Report Service- Provides functionality for capturing and reporting

metrics.

* These services are satisfied with COTS or Open Source software products.


Build 2 DeliverablesTools

• Design Tool- Provides functionality for designing product label

schemas using the XML Schema standard.

• Generate Tool- Provides functionality for generating product labels

from and conforming to the specific product schema.

• Validate Tool- Provides functionality for validating product labels

and product data.


Development ProgressBuild 2a – New and Modified Features

• Introduced the Search Service and the initial implementation of the PDS Search Protocol.

• Modified the Harvest Tool with respect to association handling and representation in the Registry Service.

• Updated the Validate Tool with the latest set of schemas for Build 2a.

• Updated the Registry User Interface to expand support for viewing additional object types.


Topics

• Overview




• Next Steps


Build 2 Deployment

• Engineering Node- Deployment of the core components replacing the

central catalog.• Will run side-by-side for six months.

- Existing search interfaces reworked to utilize the PDS4 infrastructure.

• Discipline Nodes- Deployment of the core components at the Nodes

will be phased with the Build 2b delivery for testing.- Once deployment and testing is complete, Nodes can

begin registration of PDS3 products with the Harvest Tool.


Build 2 DeploymentEngineering Node


Build 2 DeploymentDiscipline Node


PDS3 Support

• The Harvest Tool supports both PDS4 registration and PDS3 registration.- PDS3 support consists of converting PDS3 labels into

PDS4 proxy labels.

- Registered for tracking and reporting purposes.

- Will be replaced when the corresponding PDS3 data set is migrated to PDS4.

• The current Catalog Ingest Tool is updated to convert the catalog files to context products and register them with the registry.


PDS3 Support cont.


Topics

• Overview




• Next Steps


Build 2 Liens

• Finalize implementation of the Catalog Ingest Tool supporting ingestion of PDS3 catalog files into the PDS4 registry.

• Add support for XML Schema 1.1 to the Harvest and Validate Tools.

• Upgrade the Data Search interface to remove the dependence on the PDS3 catalog database.


Next Steps

• Planned for Build 2b- Address any software liens required to achieve

operational objectives for Build 2.- Continue development of capabilities targeted for

Build 3 deployment.

• Planned for Build 3- Implement data-related tools for transformation and

visualization.- Integrate above functionality into PDS search and

distribution services.- Replace any existing software (and its functionality)

that is dependent on the PDS3 catalog database.


Questions/Comments

system design/implementation and support for build 2 pds management council face-to-face mountain...

Documents

catalog system

data products

pds harvest tool

data producers

tracking data

pds standards

data delivery

system of registries