persistent management of distributed data reagan w. moore general atomics, inc

National Partnership for Advanced Computational InfrastructureSan Diego Supercomputer Center

Persistent Management of Distributed Data

Reagan W. MooreGeneral Atomics, Inc.

San Diego Supercomputer Centermoore@sdsc.edu

http://www.npaci.edu/DICE/

Topics

• Data management systems– Data collections, digital libraries

• Distributed data management– Data grids

• Persistent data management– Persistent archives

• Common infrastructure for data management

Data Collections

• Define the context for describing a collection of digital entities– Context specified by metadata attributes– Provenance, origin of the digital entities– Administrative, location of the digital entities– Technical, purpose of the digital entities

• Support organization of attributes as hierarchy of sub-collections

Digital Libraries

• Provide services on the data collection– Ingestion, loading of attribute values– Extensibility, definition of new attributes– Discovery, queries on attributes– Browsing, hierarchical listing– Presentation, formatting specified data models

Data Grids

• Manage data in a distributed environment– Logical name space, provide global identifier– Data access, storage system abstraction– Replication, disaster back up– Uniform access, common API across file

systems, archives, and databases– Single sign-on, authenticate across

administration domains

Persistent Archives

• Manage technology evolution– Storage system abstraction, support data

migration across storage systems– Information repository abstraction, support

catalog migration to new databases– Logical name space, support global persistent

identifier

Storage Resource Broker

• Integration of collection-based management of digital entities, with– Remote data access through storage system

abstraction– Catalog access through information repository

abstraction– Automation through collection-owned data

Capabilities

• Support legacy systems

• Integrate archives with file systems

• Share distributed data

• Maintain persistent collection

• Control data access

Digital Entities• Digital entities are “images of reality”,

made of– Data, the bits (zeros and ones) put on a storage

system– Information, the attributes used to assign

semantic meaning to the data– Knowledge, the structural relationships

described by a data model

• Every digital entity requires information and knowledge to correctly interpret and display

Digital Entities

• Files– Text documents, images, spread sheets, binary

• URLs

• Database query commands

• Databases

Digital Entities

• Register digital entities into a catalog

• Assign metadata to describe each digital entity

• Separate management of the associated data bits from management of the metadata

• Support manipulation of each digital entity data type

Technology Management

Old Storage System

New Operating System

Old Application

Digital Object

Old Display System

Wrap Storage System Wrap Display System

Migrate Encoding Format

Preservation of Data

• Migration – Preserve the data bits– Preserve the digital entity name– Preserve the information and knowledge

content for presentation by new applications

Migration Advantages

• By migrating the digital entity encoding format to new standards, more sophisticated technologies can be applied to express the information and knowledge content inherent in collections of digital entities.

• Requires the ability to associate data model with digital entity

Uniform API

• Provide common access semantics

• Map from the interface preferred by your application to the interfaces required by legacy storage systems

Unix Shell

Java, NTBrowsers

WebWSDL

GridFTP

SDSC Storage Resource Broker & Meta-data CatalogCommon APIs

ArchivesHPSS, ADSM,UniTree, DMF

DatabasesDB2, Oracle,

Postgres

File SystemsUnix, NT,Mac OSX

Application

AccessAPIs

Servers

Storage AbstractionCatalog Abstraction

DatabasesDB2, Oracle, Sybase

C, C++, Libraries

Logical Name Space

LatencyManagement

DataTransport

MetadataTransport

Consistency Management / Authorization-AuthenticationPrimeServer

Linux I/O

DLL /Python

Discovery Transparencies

• Naming transparency - find a data set without knowing its name– Map from attributes to a global file name

• Location transparency - access a data set without knowing where it is– Map from global file name to local file name

• Access transparency - access a data set without knowing the type of storage system– Federated client-server architecture

Unix Shell

Java, NTBrowsers

WebWSDL

GridFTP

SDSC Storage Resource Broker & Meta-data CatalogTransparencies

Postgres

Application

AccessAPIs

Servers

C, C++, Libraries

Logical Name Space

LatencyManagement

DataTransport

MetadataTransport

Linux I/O

DLL /Python

Persistent Collection

• Maintain authenticity– Authenticate all accesses– Assign roles for access control lists (curation,

write, annotate, read)– Manage audit trails of all operations

• Collection-owned data– All accesses through the data management

system

Unix Shell

Java, NTBrowsers

WebWSDL

GridFTP

SDSC Storage Resource Broker & Meta-data CatalogPersistency

Postgres

Application

AccessAPIs

Servers

C, C++, Libraries

Logical Name Space

LatencyManagement

DataTransport

MetadataTransport

Linux I/O

DLL /Python

Preservation(Similar requirements to a data grid)

• Name transparency– Find a file by attributes (map from attributes to global name)

• Location transparency– Access a file by a global identifier (map from global to local

file name)

• Access transparency– Use same API to access data in archive or file cache

• Authenticity– Disaster recovery, replicate data across storage systems– Audit and process management

Unix Shell

Java, NTBrowsers

WebWSDL

GridFTP

SDSC Storage Resource Broker & Meta-data CatalogPreservation

Postgres

Application

AccessAPIs

Servers

C, C++, Libraries

Logical Name Space

LatencyManagement

DataTransport

MetadataTransport

Linux I/O

DLL /Python

Convergence of Technologies

• Data grids as basis for distributed data management– Federation of distributed resources– Creation of logical name space to automate discovery

• Distributed data collections– Discovery based on attributes– Distributed data storage systems

• Digital libraries– Development of services for manipulating, viewing data

• Persistent archives– Management of technology evolution

Data Naming Ontologies

Concept space Discipline concepts

Data grid Global Identifier

Collection Discipline attributes

Archive / file systems Local file name

Data model Attributes that describe data structure

Knowledge Creation Roadmap

• Knowledge syntax (consensus)– RDF, XMI, Topic Map

• Knowledge management (recursive operations)– Oracle parallel database

• Knowledge manipulation (spatial/procedural rules)– Generation of inference rules and mapping to data models

• Knowledge generation (scalable inference engine)– Application of inference rules in inference engine

Knowledge Based Data Grid Roadmap

AttributesSemantics

Knowledge

Information

Ingest Services

Management AccessServices

(Model-based Access)

(Data Handling System - SRB)

InformationRepository

Attribute- based Query

Feature-basedQuery

Knowledge orTopic-Based Query / Browse

KnowledgeRepository for Rules

RelationshipsBetweenConcepts

FieldsContainersFolders

Storage(Replicas,Persistent IDs)

Further Information

http://www.npaci.edu/DICE

persistent management of distributed data reagan w. moore general atomics, inc

Documents

general atomics aeronautical systems, inc. · general...

nixon reagan

reagan years

reagan ranch

predator b - general atomics aeronautical systems ·...

atomics, locks, and tasks · atomics, locks, and tasks...

reagan reading

chasing away rats: semantics and evaluation for relaxed...

mono atomics

general atomics phy proposal to...

progress in target metrology at general atomics

atomics - university of north texas

ronald reagan

general atomics aeronautical systems, inc. · general...

1 preparing graduate students for non-academic careers dr....

reagan library topic guide assassination attempt on ... ·...

mq-9 reaper/predator b - general atomics · pdf filemq-9...

national partnership for advanced computational...

seaguardian - general atomics aeronautical systems ·...

atomics intei^ational - digital.library.unt.edu