copyright jet propulsion laboratory1 object oriented data technology dan crichton steve hughes...
Post on 27-Dec-2015
220 Views
Preview:
TRANSCRIPT
Copyright Jet Propulsion Laboratory 1
Object Oriented Data Object Oriented Data TechnologyTechnology
Dan Crichton
Steve Hughes
JPL/Caltech
CCSDS XML Workshop
Oxford, England
April 3-5, 2002
Copyright Jet Propulsion Laboratory 2
Overview
Funded in 1998 by the NASA Office of Space Science to research technology in middleware and distributed object computing architectures to enable interoperability of data management systems
Copyright Jet Propulsion Laboratory 3
OODT Focus Areas Information Architectures
– Metadata Management– Data Languages (i.e. XML)
Distributed Architectures– Messaging– Distributed Object Computing– P2P Architectures
Development of an architecture for distributed information management and retrieval
Application to the Planetary Data System to support Mars Odyssey Data Distribution
Copyright Jet Propulsion Laboratory 4
Middleware
Web, GUI, console,2D/3D visualization,immersionproduct data,
real-time data
Applications
s/w, h/w,models
Middleware
Dat
aU
ser Interface
N. Lamarra
Copyright Jet Propulsion Laboratory 5
Middleware Review History
– Pre-web: IDE (dev), client/server (apps)– Post-web: n-tier c/s, thin/thick, frameworks, …– Communications: RPC, MOM, Web , …– Applications: Desktop and Remote, …– Architecture: Open Reference Model, HLA, TOGAF, …
Standards– O/S: POSIX, Unix, Windows– Communications: TCP/IP, OSI, DCE/ONC, HLA/RTI, HTTP– DOC: CORBA/COSS, DCOM/COM+, Java RMI– Data access: O/JDBC, EJB persistence, STEP, HTML, XML– Security: SSL, SSH, SHTTP, IPSec, PKI, CDSA, BioAPI, etc.– Directory: X.500, LDAP– Systems Management: CIM, SNMP, CMIS/CMIP– GUI: X11, CDE, Java-Swing
Copyright Jet Propulsion Laboratory 6
OODT Principles
Location Independence Information Hiding Model driven architecture Scalable, Extensible Client APIs for locating and accessing
distributed data products
Copyright Jet Propulsion Laboratory 7
Model Driven Architecture
Decouple solutions from platform specific technology
Allows data architecture and technology architecture to evolve independently
The technology architecture supports the data architecture (i.e. it supports the model)
Apply architecture to different problem spaces
Copyright Jet Propulsion Laboratory 8
OODT Architecture
Address two key architectural focuses– Technology Architecture
• Basic communication between geographically distributed data systems
• Framework for plugging in new data systems
– Data Architecture• Standard method for describing data (profile)
• Standard method for exchanging data (XMLQuery)
Copyright Jet Propulsion Laboratory 9
OODT Standards The architecture for accessing distributed resources is based on
the following standards:– W3C Extensible Markup Language (XML) 1.0 (Second Edition)
(Recommendation 6 October 2000)– W3C HTML 4.01 Specification (Recommendation 24 December
1999)– W3C XHTMLT 1.0: The Extensible Hypertext Markup
Language (Recommendation 26 January 2000)– ISO/IEC 11179 - Specification and Standardization of Data
Elements, Parts 1-6, ISO/IEC specification, http://www.iso.ch/iso.
– Dublin Core Metadata Initiative. The Dublin Core Element Set Version 1.1, Dublin: DMCI, July 1999.
Copyright Jet Propulsion Laboratory 10
OODT Technology Implementation
Implemented in Java Data layer implemented with XML Uses CORBA for remote object invocation
– We use JacORB SSL for data encryption Uses a standardized XML DTD (XMLQuery) as the
messaging format (See http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd)
Supports a variety of client access methods– Java API– HTTP– C++ API
Copyright Jet Propulsion Laboratory 11
Why XML? XML doesn’t provide a “silver bullet,” but it does allow us
to refocus the problem on metadata– Metadata is a key to interoperability
XML is language neutral Allows the designer to separate the data and the transport
(re: CORBA vs XML-over-CORBA)– Transport mechanism and data are not tied together
• Could be XML/HTTP• Simpler deployments• Simpler interfaces• Allows technologies to grow and change independently
Real value of XML is providing a content language for documenting the information architecture
Copyright Jet Propulsion Laboratory 12
XML and CORBA XML over CORBA/IIOP module jpl {
module user { interface UserManager { string do(string xml);
}; };};
<transaction> <findUser> <user> <surname>Doe</surname> </user> </findUser></transaction>
CORBA method
module jpl { module user { interface UserManager { User findUser(string
name); }; interface User { String getName(); }; };};
Copyright Jet Propulsion Laboratory 13
OODT Data Architecture
Use Profiles to describe data resources– System Profiles– Dataset Profiles– Product Profiles
A profile is just a metadata description of the resource defined by an XML DTD (see oodt/dtd/prof.xml)– Uses Dublin Core – Uses ISO 11179
Copyright Jet Propulsion Laboratory 14
What is a profile?
• Profiles describe and provide the location for anything of interest– Things of interest (data and system resources)
• Data Systems, Data Sets, Web Sites, etc
– Written as XML documents– Grouped
• Data Product profiles grouped by data set• All others grouped into single database
– Provide sufficient information to determine if the resource can resolve a user query
Copyright Jet Propulsion Laboratory 15
Profile Overview Implementation
A profile includes a pointer to the resources
A profile has three main sections. The profAttributes (Profile Attributes) section describes the
profile. The resAttributes (Resource Attributes) section describes
the resource by using a common set of keywords (Dublin Core)
The profElement (Profile Element) section describes the resource by using domain specific attributes (PDS keywords)
Copyright Jet Propulsion Laboratory 16
Profile DTD<!ELEMENT profiles (profile+)>
<!ELEMENT profile (profAttributes, resAttributes, profElement*)>
<!ELEMENT profAttributes (profId, profVersion*, profTitle*, profDesc*, profType*, profStatusId*, profSecurityType*, profParentId*, profChildId*, profRegAuthority*, profRevisionNote*, profDataDictId*)>
<!ELEMENT resAttributes (Identifier, Title*, Format*, Description*, Creator*, Subject*, Publisher*, Contributor*, Date*, Type*, Source*, Language*, Relation*, Coverage*, Rights*, resContext*, resAggregation*, resClass*, resLocation*)>
<!ELEMENT profElement (elemId*, elemName, elemDesc*, elemType*, elemUnit*, elemEnumFlag*, (elemValue | (elemMinValue, elemMaxValue))*, elemSynonym*, elemObligation*, elemMaxOccurrence*, elemComment*)>
Copyright Jet Propulsion Laboratory 17
?xml version="1.0"?>
..
<!DOCTYPE profiles SYSTEM “http://ginger.fhcrc.org/dtd/prof.dtd">
<profiles> <profile>
<profAttributes>
<profId>1.1.1.1.102</profId>
<profType>profile</profType>
<profStatusId>ACTIVE </profStatusId>
</profAttributes>
<resAttributes>
<Identifier>EDRN_MOFFITT_PRODUCT_SERVER</Identifier>
<Title>EDRN H. Lee Moffitt Cancer Center Product Server</Title>
<Format>text/html</Format>
<Description>A product server providing access to a database of clinical sample information for the Early Detection Research Network (EDRN) at the H. Lee Moffitt Cancer Center.</Description>
<Language>en</Language>
<resContext>NIH.NCI.EDRN.MOFFITT</resContext>
<resAggregation>granule</resAggregation>
<resClass>system.productServer</resClass>
<resLocation>iiop://oodt.jpl.nasa.gov:10000/NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER</resLocation>
</resAttributes>
<profElement>
<elemId>STUDY_PARTICIPANT_ID</elemId>
<elemName>STUDY_PARTICIPANT_ID</elemName>
<elemDesc>EDRN Participant ID</elemDesc>
<elemType>CHARACTER</elemType>
<elemEnumFlag>F</elemEnumFlag>
</profElement>
<…>
Copyright Jet Propulsion Laboratory 18
Adopt an XML Query Structure for Messaging
Support queries across distributed resources
Provide a standard for inter-communication between heterogeneous applications and computing environments
Describe both queries and results
Copyright Jet Propulsion Laboratory 19
XMLQueryImplemented in XML at http://oodt.jpl.nasa.gov/dtd/xmlquery.dtd
Defined as followsThe query description
The resultsResult 1
Result Header
Result Data
Result 2Result Header
Result Data
...
Result NResult Header
Result Data
Final results are aggregated from all profile/product servers
Copyright Jet Propulsion Laboratory 20
XML Query Example (1 of 2)<query> <queryAttributes> <queryId>1.2.3.4…</queryId> <queryTitle>OODT_XML_QUERY Query Example</queryTitle> <queryDesc>Query for TARGET_NAME = MARS</queryDesc> <queryType>QUERY</queryType> <queryStatusId>ACTIVE</queryStatusId> <querySecurityType>UNKNOWN</querySecurityType> <queryRevisionNote>2000-05-12 JSH V1.2 Updated for new prof.dtd</queryRevisionNote> <queryDataDictId>1.2.3.5…</queryDataDictId> </queryAttributes> <queryResultModeId>ATTRIBUTE</queryResultModeId> <queryPropogationType>BROADCAST</queryPropogationType> <queryPropogationLevels>N/A</queryPropogationLevels> <queryMaxResults>100</queryMaxResults> <queryResults>0</queryResults> <queryKWQString>TARGET_NAME = MARS</queryKWQString>
Copyright Jet Propulsion Laboratory 21
XML Query Example (2 of 2)<querySelectSet></querySelectSet> <queryFromSet></queryFromSet> <queryWhereSet> <queryElement> <tokenRole>elemName</tokenRole> <tokenValue>TARGET_NAME</tokenValue> </queryElement> <queryElement> <tokenRole>LITERAL</tokenRole> <tokenValue>MARS</tokenValue> </queryElement> <queryElement> <tokenRole>RELOP</tokenRole> <tokenValue>EQ</tokenValue> </queryElement> </queryWhereSet> <queryResultSet></queryResultSet></query>
Copyright Jet Propulsion Laboratory 22
OODT Components to Support the Architecture
Name Server Query Server Archive Server Profile Server Product Server Server Manager
Copyright Jet Propulsion Laboratory 23
OODT Software Components
Name Server– CORBA name server that maps an object to a location (i.e.
ern:NIH.NCI.EDRN.MOFFITT.PRODUCT_SERVER can be communicated at edrn.moffitt.edu:7877)
– All implementations of CORBA, DCE, DCOM, SOAP, etc use a name server
– Only needs to be run once at a central site
Query Server– Implemented by OODT – Multi-threaded server for managing queries to distributed servers– Aggregates results from multiple servers– Only needs to be run once at a central site
Copyright Jet Propulsion Laboratory 24
Archive Server
Provides a server framework for– Adding products remotely to a generic archive– Retrieving products remotely from a generic archive
A generic data model that is driven by the metadata model– Captures a catalog of archived products using well
defined data elements Uses XML to label data products Extract Profiles of archived products
Copyright Jet Propulsion Laboratory 25
Profile Server Profile Server
– Used to identify the existence and location of resources in a distributed data environment
– Two types of profile servers• Dynamic Profile Server
– Front ends an existing database catalog to extract the resources and format them as profiles
– Implemented as a Java Interface class
• Static Profile Server– Manages its own local resources
– XMLQuery is the messaging structure to query and return profiles
– As many profile servers can be run as needed to scale the system. Typically one per catalog.
Copyright Jet Propulsion Laboratory 26
Profile Server Overview
• Profile Servers serve profiles– Allow search and retrieval of profiles– Retrieves from profile databases
• Static profiles stored as XML documents• Dynamic profiles generated from information stored in
databases
– Can be distributed for local management and scalability at a site
Copyright Jet Propulsion Laboratory 27
Profile Server Requirements
1) A profile server shall search and retrieve profiles from a profile database
2) For search, a profile server shall allow any profile attribute as a query constraint. Profile attributes include those from the profile element, resource attribute, and profile attribute sections of the profile document.
3) For retrieval, a profile server shall return matching profiles. The user can request the complete profile or any subset of the profile.
Copyright Jet Propulsion Laboratory 28
Profile ServerXML Interface
1) Input: XMLQUERY – A profile server accepts an XMLQUERY XML document as input. User query constraints have been encoded into the WHERE clause of the XMLQUERY. The requested profile attribute results have been encoded into the SELECT clause.
2) Output: XMLQUERY – A profile server inserts the set of matching profiles in the RESULTS clause of the XMLQUERY.
Copyright Jet Propulsion Laboratory 29
Profile ServerServlet Interface
• User calls optional servlet using “DIS” query• Servlet creates XMLQUERY and calls predetermined
profile server• Results returned in XMLQUERY• Query Server passes parallel query to each product
server that has a match
htpp://ginger.fhcrc.org/servlet/jpl.oodt.servlets.Profile_Servlet?
keywordQuery={CANCER_SPECIMEN_TYPE = BLOOD
&object= NIH.NCI.EDRN.DMCC.PROFILE_SERVER
&mimeType=XML
&mimeType=*/*}
Copyright Jet Propulsion Laboratory 31
Product Server
Product Server– Used to return a product using the XMLQuery
messaging structure– Implemented as a Java Server– Contains one or more query handlers
• Query handler implemented as a Java interface class
• Query handlers loaded at server start time
• Defined in a local configuration file
– Can use any method to access and return data
Copyright Jet Propulsion Laboratory 32
Product Server Implementation The Product Server plugs into the OODT framework and
manages the “handshake” between the data system and the OODT system.
Extensible by dynamically loading objects at runtime which are specific to the data system model
Queries and results are passed using an OODT XML Query structure
Product Server
Flat file accessor
Unusual accessor
Query
Result
Generic Server Implementation Class
FileSys
Database
Copyright Jet Propulsion Laboratory 33
Profile Server
DistributedProfileServers
Product Servers for Planetary Data System
HTTP, IIOP, Java, C++
APIs
ASU ThemisProducts
jdbc:// jdbc://
Data Source InterfaceFor
Dynamically Loaded Query Handlers
Java Server Framework
PDS.ASU.THEMIS.PRODUCT_SERVER
Mars OdysseyImaging Products
Java Server Framework
PDS.IMAGING.PRODUCT_SERVER
XMLQUERY
Data Source InterfaceFor
Dynamically Loaded Query Handlers
HTTP, IIOP, Java, C++
APIs
XMLQUERY
Copyright Jet Propulsion Laboratory 34
Layered OODT Product Server Architecture
PC/Sun,etc Hardware
Unix/Windows operating system
Java
OODT
CORBA/SSL
Local Product serversProvided by JPL
Provided by hosting site
Stability
Copyright Jet Propulsion Laboratory 35
Server Manager
Server Manager– Distributes software– Manages distributed software– Remote Management and Debugging – Automatic process management– APIs to allow tools to inquire about server
status (i.e PDS Themis Node at ASU is up or down)
Copyright Jet Propulsion Laboratory 36
Challenges with Distributed System Integration
Off site development– Debugging by telephone
New databases and database drivers– Inexperience led to programming errors
Institutional Challenges– Finicky configurations– Mysterious network issues
Copyright Jet Propulsion Laboratory 37
Possible Solutions
Remote PC control– Unacceptable security risk at some sites
Mandate Unix– Lack of IT experience at member sites– Retooling time too great
Put remote management into the product server itself
Copyright Jet Propulsion Laboratory 38
Server Manager
Manages servers– Server manager runs at each site
Client/server implementation– XML-RPC with HTTP authentication
One-click installation– Well, almost: install Java first
Capture Log Files Perform restarts
Copyright Jet Propulsion Laboratory 39
Remote Control Application
Authenticates with a server manager Enables remote administration of server
processes– View log files and process output– Set Java runtime parameters– Update files– Define, start, stop processes– Execute upgrade scripts
Copyright Jet Propulsion Laboratory 41
Software Distribution Environment
Installation– First, you install Java– Windows users download and run one self-
installer•pds-setup.exe• Displays typical Windows-style wizard installer
– Unix/Mac users download a tarball and follow instructions in a README file
Copyright Jet Propulsion Laboratory 42
OODT Configurations
oodtrc.xml– Local configuration file that contains
• Server names
• List of profile servers
• Logging methods
• Timeout methods
• Database configuration parameters
• …
Copyright Jet Propulsion Laboratory 43
OODT Applied to PDS
Provides a common messaging infrastructure for PDS
Provides data abstraction and location independence for PDS Nodes– Common tools can look across the PDS sites – Heterogeneous implementations are standardized
Uses the PDS Data Dictionary Uses existing PDS databases and catalogs
Copyright Jet Propulsion Laboratory 44
Generic Architecture
Query Server
Profile Server
Site 3Product Server
XMLQuery
XMLQuery
XMLQuery
XM
LQ
ue
ry
XM
LQ
ue
ry
ProfileServer
Que
ryC
lien
tWeb
se
rver
Plu
gin
s
Web
Ser
ver
XMLQuery
Site 1 DB
Site 2 DB
Site 3 DB
Web I/F
Desktop I/F
XMLQuery
XMLQuery
XMLQuery
XMLQuery
Name Server
ClientEnvironment Distributed Repositories
Site 2Product Server
Site 1Product Server
Copyright Jet Propulsion Laboratory 45
Profile/Location Server
Profile DB
Default DataSet Browsers
Custom DataSet Browsers
Middleware
PDS-DistributionDelivery - D01Architecture
Single Point of EntryData Set View
Data Set SelectionPage
Data SetCatalog
Other NodeResources
ProductServer
GRS GRS RS
ProfileServer
RS
ProductServer
THEMIS THEMIS
ProductServer
SPICE
MARIE
Custom DataSet BrowsersCustom Data
Set Browsers
Copyright Jet Propulsion Laboratory 46
OODT Projects
PDS D01 will use OODT to support Mars Odyssey Distribution (October 2002)
PDS D02 will rollout OODT to all of PDS OODT building a data architecture for the
National Cancer Institute’s (NCI) Early Detection Research Network integrating up to 40 sites by 2005
OODT Supporting development of a JPL Institutional Data Architecture with Data Management Services
Copyright Jet Propulsion Laboratory 47
OODT References
OODT Overview– http://oodt.jpl.nasa.gov
OODT Papers– http://oodt.jpl.nasa.gov/doc/papers
top related