c. huc/cnes, d. boucon/cnes-silogic, d.m. sawyer/nasa/gsfc, j.g. garrett/nasa-raytheon...

C. Huc/CNES, D. Boucon/CNES-SILOGIC,D.M. Sawyer/NASA/GSFC, J.G. Garrett/NASA-Raytheon

Producer-Archive Interface MethodologyProducer-Archive Interface MethodologyAbstract StandardAbstract Standard

PAIMAS

SpaceOps, Montreal, 2004-05-15 Producer-Archive Methodology 2

ContentsContents

Why a new standard?• Space data archiving• Needs for standardization• What standardization will bring• Abstract standard status

Proposed methodology• Context - description• The 4 phases

Specialization Aid Tool for the use of the Methodology Towards an implementation standard Conclusions


Why a new standard?Why a new standard?Space data archiving: archive typesSpace data archiving: archive types

Telemetry Operation control

Payload control

Mission exploitation

Missionarchives

Mission control system

Scientific multi-missiondata centers: long-term archives


Data Producers: • the operating systems of each mission• the mission processing and archiving centres• the scientific teams in the laboratories

Archive• specialized independent organizations (CDS)• specialized services within the agencies (NSSDC)• services common to several organizations (CDPP)

a Producer-Archive Project is a set of activities and the means used by the Information Producer as well as the Archive to ingest a given

set of information into the Archive.

Why a new standard?Why a new standard?Space data archiving: key roles and playersSpace data archiving: key roles and players


The relations between archives and data Producers are rarely simple and easy:• nonconformity of received data• unclear and imprecise definition of the data to be delivered,• failure to meet delivery schedule,• late detection of errors in archived data,• non-management of modifications

==> Can be detrimental to archived information quality and the cost of the operation.

Ever increasing diversity of the producersData complexityEach project develops its own methodology on the

basis of a process that is roughly the same from one project to another

==> Work duplicated, no generality, excessively high costs, etc.

Why a new standard?Why a new standard?Needs for standardization: problemsNeeds for standardization: problems


PAIMASGeneric abstract

standard

Communityabstract standard

PAIMASImplementation

standard

Approved by the CCSDS

Development in progress

Why a new standard?Why a new standard?Needs for standardization: multiple standardsNeeds for standardization: multiple standards


Ease some typical problems (as noted earlier) Make it easier to follow schedules and commitments Help with human resources turnover Anticipate new problems Address the legal aspects early Increase data quality of archive holdings Have weight and credibility for adoptionAnd finally, save time (and money)

Why a new standard?Why a new standard?What standardization will bringWhat standardization will bring


Today: CCSDS standard (blue book) and standardization with ISO in progress

Various communities have helped produce this document :

• space agencies (CNES, NASA, ESA, …)

• archivists (NARA, …) and

• librarians (BNF -French National Library, RLG-Research Library Group).

Why a new standard?Why a new standard?Abstract standard statusAbstract standard status


ContentsContents



Specialization Aid Tool for the use of the Methodology Towards an implementation standard Conclusion


Producer

Consumer

Queries

Results sets

Orders

Submission

Information

Packages

Dissemination Information Packages

OAIS

Archival Information

Packages

MethodologyMethodologyContext: OAIS environmentContext: OAIS environment


AIP : Archival Information Package, SIP : Submission Information Package, DIP : Dissemination Information package

M A N A G E M E N T

I n g e s t

D a t aM a n a g e m e n t

S I P

A I PD I P

q u e r i e s

r e s u l t s e t sA c c e s s

PRODUCER

CONSUMER

D e s c r i p t i v eI n f o

A I P

o r d e r s


A r c h i v a lS t o r a g e

A d m i n i s t r a t i o n

P r e s e r v a t i o n P l a n n i n g

MethodologyMethodologyContext: PAIMAS focusContext: PAIMAS focus


AIP : Archival Information Package, SIP : Submission Information Package, DIP : Dissemination Information package

M A N A G E M E N T

I n g e s t

D a t aM a n a g e m e n t

S I P

A I PD I P

q u e r i e s

r e s u l t s e t sA c c e s s

PRODUCER

CONSUMER


A I P

o r d e r s


A r c h i v a lS t o r a g e

A d m i n i s t r a t i o n

P r e s e r v a t i o n P l a n n i n g

Ingest

MethodologyMethodologyContext: PAIMAS focusContext: PAIMAS focus


The archive project is broken down into 4 main phases:

• Preliminary Phase,

• Formal Definition Phase,

• Transfer Phase,

• Validation Phase.

Each phase has extensive action tables.

Specification for a community.

MethodologyMethodologyDescriptionDescription


Data ready to archive

MethodologyMethodologyThe phases: relationshipsThe phases: relationships

Preliminary Agreement

DictionaryFormal modelSubmission Agreement

Transferred object files

Validation agreement

Ph

ase

obje

ctiv

e

Preliminary Phase

Formal Definition Phase

Transfer Phase

ValidationPhase

Anomalies

Validate

the

transfe

rred

objects

• Iden

tifica

tion a

nd pre

limin

ary

agre

emen

t

Defi

ne th

e in

form

atio

n

to

be

arch

ived

•reso

urces

est

imat

ion

•Neg

otia

te th

e Su

bmiss

ion

Dev

elop

agre

emen

t (dat

a to

be

deliv

ered

, com

plem

enta

ry

ele

men

ts, s

ched

ule)

•Act

ual t

rans

fer o

f the

dat

a

Act

ual tr

ansf

er o

f the

ob

ject

s


Preliminary Phase


MethodologyMethodology Preliminary phase: context Preliminary phase: context

Archive

Producer


First contact

Preliminary definition,feasibility and assessment

Establishment of apreliminary agreement

MethodologyMethodologyPreliminary phase: sub-phasesPreliminary phase: sub-phases

Information to be archived, Quantification, Legal andcontractual aspects, permanent impact on the Archive,Summary of costs, etc.

Id Preliminary phase: quantification Involves

P-19 Estimate the data volume to be transmitted to the Archive Producer

P-20 Assess the permanent data volume to store Archive

P-21 Assess the storage capability need for the ingest process Archive

P-22 Assess the associated costs Archive

Action table

Description


Formal Definition Phase


Dictionary

Data Model

Submission Agreement

MethodologyMethodologyFormal Definition Phase: contextFormal Definition Phase: context


Organization of the FormalDefinition Phase

Formal definition

Submission Agreement

Id Formal Definition Phase: Definition of Transfer Conditions Involves

F-14 Define the communication procedures (digital network, protocols,media, etc.)

Producer and Archive

F-15 Define the Packaging Information of delivered objects (in whatform the data is delivered)


F-16 Define a transfer session (functional and time-related structure ofthe transfer of digital objects)


F-17 etc

MethodologyMethodologyFormal Definition Phase: sub-phases and action tableFormal Definition Phase: sub-phases and action table

Creation of a data dictionary, definition of transferconditions, delivery schedule, etc.


Transfer PhaseActual transfer of the objects:• carry out the transfer test• manage the transfer

Data

Model of object files to deliver

Schedule


MethodologyMethodologyTransfer PhaseTransfer Phase


Validation Phase

Validate the transferred objects:• carry out the validation test• manage the validation


Data ready to archive

Anomalies

MethodologyMethodologyValidation PhaseValidation Phase

Producer

Validation

acknowledgement


ContentsContents





Adapt the generic standard to a particular community (which can range from an international organization to a simple archive service)

Steps involved to define a community standard• terminology,

• data dictionary and information model,

• standards,

• common tools.

Analyze each action of the generic standard (add actions if necessary)

SpecializationSpecialization


Example (CDPP -Plasma Physics Data Center- Archive)

F-4: Define the formats, coding rules, and standards to be applied for the objects to be delivered

Generic Action (Methodology)

Specific Action (CDPP)

Apply the ‘specification of products for delivery to the CDPP’ including

Use the Data Description Language EAST Specification defined by the CCSDS for the syntactic data description

Use the CCSDS DEDSL-Data Entity Dictionary Structure Language- —XML/DTD Syntaxfor the semantic data description

Use the CCSDS time code format for the date and time coding

Etc.

Specialization: ExampleSpecialization: Example


ContentsContents





A methodology application and archive project aid tool would be a real advantage for both the Producer and the Archive.

First mockups produced

Aid ToolAid Tool


Aid ToolAid Tool


ContentsContents





PAIMASGeneric abstract

standard

Implementation standard

Methodology application aid

Languages

Tools

Communitystandard

Archive project

Towards an implementation standardTowards an implementation standardContextContext


• generic data model– transfer data model (generic objects)

– transfer data dictionary (generic descriptors)

– community dictionary

• diagram of data to transfer for the

archive project

• transfer sequencing and chronology

• transfer session

Implementation standard

Languages

Tools

XML

XML Schema

UMLXMLScheme

DEDSL

DTDXMLScheme

XFDU

Towards an implementation standardTowards an implementation standardApproachApproach


Towards an implementation standardTowards an implementation standardRelationshipsRelationships

CLUS-CIS-CODIF-Jm

CLUS-CIS-CODIF-EAST

CODIF

CIS

HIA

CLUS-CIS-HIA-EAST

CLUS-CIS-HIA-Jn

CLUSTER


Delivery plan

• all the elements to be delivered

• easy to understand from the Producer side

• expression of constraints

Towards an implementation standardTowards an implementation standardSchemaSchema


ConclusionConclusion

PAIMAS identifies: • the phases in the process of transferring information,

• the objective of the phases,

• the actions that must be carried out,

• the expected results.

PAIMAS is a basis:

• for the identification of standards and implementation guides,

• for identification and development of a set of software tools.

c. huc/cnes, d. boucon/cnes-silogic, d.m. sawyer/nasa/gsfc, j.g. garrett/nasa-raytheon...

Documents

new standard

new problems

space data archivingneeds

archived data

ccsds standard blue

space agencies cnes

information quality

submission information