a virtual distributed database model for creating a database federation

A Virtual Distributed Database Model for Creating a Database Federation

Jaspreet Singh PahwaPhD Year 2

School of Computer Science

Cardiff University

Overview

• Motivation

• Hypothesis

• Product Class

• Product Class Database

• Grid Enabled Multiple Database Search Service (MDSS)

• MDSS Federation

• Conclusions and further work

Motivation

• Information held by large number of data sources in a Wide Area Network.

• Access to such information held by large number of organisations.

• Information processing with aid of independent mechanisms, and

• Sharing of information with those who need it.

Hypothesis

• Enabling product definition infrastructure.• Creating standard product definition using the infrastructure.• Subscription of product definitions and product creation by

product suppliers.• Building a virtual distributed database using the Grid

infrastructure. • Participation of product suppliers in the virtual distributed

database as part of a federation.• Advertisement of products in the virtual distributed database by

Product Suppliers. • Product search using product definition and the Grid.

The Product Class

• A standard product definition for providing product information.

• Acts as a template and provides meta information for creating actual product information.

• Is made up of a number of specifications conforming to different specification types.

• Can be used by Product Suppliers for product advertisement in the federation.

Sub Product Class Specification

Specification Group

Specification

Sub-Specification Group

Table Spec

List Specification

Product Class Specification

Specification

Value

Table Specification

List of Column Specifications where each column specification has list of values relative to other column specification values.

List Specification

Value

Value

…

Product Class

The Product Class and its various specification types

Product Class Versioning

• Versioning of Product Class/Specifications.• New products emerge.• Existing products evolve and are assigned

more attributes.• New versions created and assigned.• Product suppliers notified.• New classes downloaded.

Product Class Subscription

Table

Table

…

Table

Table

…

XmlParse

Service

<xml> … … …</xml>

Product Class Database

Transport Layer

Transport Layer

Product Supplier Database

The Product Class Database

• A tool for creating Product Classes and Product Class Specifications.

• Enables Product Class versioning.

• Releases Product Class for subscription by Product Suppliers.

• Provides meta information for building product search criteria.

Product Class Database Structure

Product Class

Product Category Product Class Version

Product Class DefinitionCategory Hierarchy

Category

Specification List Version Specification Group Version

Table Version

List Specification

Sub Product Class Version

Specification Group

Table Specification

Sub Product Class Definition

List DefinitionSpecification Group Definition

Table Definition

Sub Product Class

The MDSS

• Multiple Database Search Service.• A database centric Grid service for accessing

and processing information from large number of data sources.

• Searches supplier data sources based on a search criteria.

• A channel for product suppliers for advertising products to a large user base.

• Built using Globus Toolkit 3.0.2 (Core).

The Search

• What to search?

- a search criteria submitted by the user.• Where to search?

- data sources that match user’s request.• What are the available resources to propagate the

search?

- the available Database Search Services in the MDSS

Search CriteriaWhere to Search? <NewDataSet> <Supplier> <IDSupplier>1</IDSupplier> <SupplierWS>http://131.251.42.40/test/categories.asmx</SupplierWS> </Supplier> …</NewDataSet>

Available GSHs <NewDataSet> <GridServiceHandle> <IDGsh>1</IDGsh> <GSH>http://131.251.42.40:8080/ogsa/services/DatabaseSearchService</GSH> </GridServiceHandle> <GridServiceHandle> <IDGsh>2</IDGsh> <GSH>http://131.251.47.110:18080/ogsa/services/DatabaseSearchService</GSH> </GridServiceHandle> …</NewDataSet>

The Architecture

Two major components:• Master Grid Service: identify resources available,

register new resources, job allocation, data parsing and data aggregation.

• Database Search Service: data search, data aggregation, data parsing, collaborative working.

- MDSS Architecture -

WS Layer WS Layer WS LayerWS Layer WS Layer WS Layer

Grid DB Search ServiceGrid DB Search Service Grid DB Search Service

Apache Axis Soap Server

<<soap messages >> <<soap messages >><<soap messages >>

Master Grid Service * Metadata Query *Job Allocation Grid Service * Collation of datasets

Single user

VO

Portal (Search Criteria Specified here)

MetadataProduct Classes

Grid Engine(based on the OGSA Model)

Supplier databases

Design Features

• Ability to register new data sources.• Ability to enhance MDSS capability by adding more

machines in the Grid cluster.• Supplier databases free of Grid Complexities.• Ability to identify data sources to be searched in real

time.• Distribute search equally among available GSHs.• New instance of Database Search Service for each

operation.

GSH Document

Parser

Supplier Document

Parser

Xml Converter

Database Search Service

Job Execution

Grid Service Manager

Data Aggregation

Master Grid

Service

OGSA Container

Master Grid Service Modules

• Information retrieval model for retrieving product information without the need of creating federated schema.

• Encompasses the MDSS System and all participating SD Systems.

• Cooperation to the centralised MDSS System for advertisement of products to VOs.

• Provides an environment for data models to compete with each other.

• Database access mechanism Grid enabled and not data sources.

• Component data models participate in the federation by providing web service interface.

MDSS Federation

The MDSS Federation

The MDSS System Database Search Job Allocation

Collation of Datasets

Single user / VO

Component Schema(Product Class)

Web Service Layer

Supplier 1

Component DBS

Component Schema(Product Class)

Web Service Layer

Supplier 2

Component DBS

Component Schema(Product Class)

Web Service Layer

Supplier n

Component DBS

Design Features

• Component DBS retain local autonomy.• Schema integration or creation of federated

or external schemas not required.• Uniform data model addresses the issue of

semantic heterogeneity.• Supports different kinds of autonomies.

Conclusions & Further Work

• MDSS Federation – a Grid enabled virtual distributed database model for product advertisement.

• Creation of downloadable product classes.• Product class subscription.• Multiple instances of Master Grid Services.• Data Streaming.• Memory Resident databases.

Questions?