- from a vision to design and implementation peter...

1

www.gridminer.org … Intelligent Grid Solutions

Knowledge Discovery Grids

Peter Brezany

Institute of Scientific Computing University of [email protected]

- from a Vision to Design and Implementation

Beijing, September 2005www.gridminer.org

Motivation

Business

Medicine

Scientificexperiments

SimulationsEarth observations

Data and dataexploration

cloud

Data and dataexploration

cloud

2


Motivation

Data

Business understanding

Dataunderstanding

DataPreparation

Modeling

Evaluation

Deployment

CRISP-DM, SPSS

ServiceProvider

ServiceProvider

ServiceProvider

Data provider

Gri

dM

iner User

Virtual Organization


Outline

MotivationGrid – Evolution TrajectoryKnowledge Discovery in DatabasesGridMiner System Developed in Vienna

Scientific and Application DriversArchitectureWorkflow ManagementGraphical User InterfaceData Integration (Mediation) ConceptsServices for Data Mining and On-Line Analytical ProcessingPerformance Issues

3


Outline (2) Towards Future Developments

Wisdom Web (Future Interconnection Environment)Current Grids and Next-Generation GridsSemantic Grids (GridMiner+)

ArchitectureWorkflow HierarchyOther Research Challenges

Conclusions GridMiner Demonstration


Media That Radically Influenced Society

Web

1500sPrinting Press

1840sPenny Post

1850sTelegraph

1920sTelephone

1930sRadio

1990s

1950sTV

20xxGrid

4


Internet vs. Grid Computing

"The Internet is aboutgetting computers to talk together;

Grid computing is aboutgetting computers to work together."

Tom Hawk, IBM's general manager of Gridcomputing


Grid Computing Concept

The Grid – a new distributedcomputing infrastructure for science, engineering, and business.

The Grid consists of physicalresources (computers, disks, networks, files, sensors, laboratoryequipments, etc.) and “middleware“software that ensures the accessand the coordinated use of such resources.

It enables communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals.

5


Grid Development PhasesBasic Grid development (Globus 1) –

metacomputingData Grid (Globus 2, DataGrid of CERN,

etc.)Semantic Grid (myGrid)Open Grid Service Architecture (Globus 3,

OGSA-DAIS) & WSRF (Globus 4)Knowledge Discovery Grid (GridMiner and

work of others)Knowledge Grids & Web Intelligence (China

& Japan, GridMiner+)


Data Exploration Issues and theGridMiner Project in Vienna

6


Knowledge Discovery Process in GridMiner

DataWarehouse

Knowledge

Cleaning andIntegration

Selection andTransformation

Data Mining

Evaluation andPresentation

OLAP

Online Analytical Mining

OLAP Queries

Data and functional resources can be geogra-phically distributed – focus on virtualization.


GridMiner (Goal) Architecture

GMMSMediation

GMPPSPreprocessing

GMDMSData Mining

GMPRSPresentation

GM DSCEDynamic Service Control

GMDTTransformation

GMOMSOLAM

GMISInformation

GMRBResource Broker

GridMiner Core

GMCMSOLAP / Cubes

GridMiner Base

GridMiner Workflow

Grid CoreServices

Security File and DatabaseAccess Service

ReplicaManagement

Grid Core

Grid Resources Data Sources

Fabric Basic GridServices

SMDSupport for Mobile Devices

GridMiner Mobility

7


Working with GridMiner

GridMinerVisible GridPart

Invisible GridPart

Database and Gridadministrator End User

administrationcommands &

queries

data explorationworkflow spec. &

results


The Grid offers..

Theoretically, the Grid can have unlimited size (the number of data and computational resources) – support for scaling up

Questions:When is it necessary to mine huge databases, as opposed to mining

a sample of the data?Should not data mining algorithms be able to take advantage of all the data that is available?

Answers:Scaling up is desirable, because increasing the size of the trainingset often increases the accuracy of induced classification models.Determining how much data to use is difficult, because the smallestsufficient amount depends on factors not known a priori.Today‘s mining techniques can have problems when data setsexceed 100 megabytes.

8


Optimizing predictive accuracyacc

ura

cy

sampled data size

100%

available data size

(qo,mo)

(qo,mo)

qi - data qualitymi - modeling method

(q0,m0)

Assumed

(qo,mo)

(qi,mi)


Pilot Application: Project EcoGRID

Waste

Air

Soil

Water

Emmision

Bio-diversity

Forests

DistributedData

DistributedData Mining

Flow Analysis

Geo-Statistic

Reporting

PopularPresen-tation

PredictionModels

DistributedApplications

…

Statistic

Common Ontology Author: Kathi Schleidt

9


Management of TBI patients

Traumatic brain injuries (TBIs) typically result fromaccidents in which head strikes an object.The treatment of TBI patients is very resourceintensive.The trajectory of the TBI patients management:

Trauma eventFirst aidTransportation to hospitalAcute hospital careHome care

All the above phases are associated with data collectioninto databases – now managed by individual hospitals.

Usage of mobile communication

devices


Collaboration of GM-Services

GMPPSPreprocessing

GMDMSData Mining

GMDISIntegration

GMPRSPresentation

Data SourcesIntermediateResult 1

IntermediateResult 2(e.g. “flat table”)

IntermediateResult 3(e.g. PMML)

FinalResult

Simple Scenario:

10


Collaboration (2)

GMDIS

GMPPS

GMPPS

GMPPS GMDMS GMPRS

GMPPS GMPPS

GMDMS

GMDMS

GMPRS

GMPRS

Complex Scenarios:

GMDMS GMPRS

GMDIS

GMPPS

GMPPS

GMCMS GMOMS GMPRSGMPPS


Workflow Models

Static Workflows Dynamic Workflows

11


Workflows

DSCE

Service A Service B

Service C

Service D

DSCL

User

Dynamic Service Control Engine (DSCE)

processes the workflow according to DSCL

Dynamic Service Control Language (DSCL)

based on XMLeasy to use

DSCE Client


Workflow language (DSCL)

composition

dscl

sequence

parallel

invoke activityID=“act2.1” …

invoke activityID=“act2.2” …

createService activityID=“act1” …

sequence

variables

act1

act2.1

act2.2

…

User´s viewConversion

to XML

12


Graphical User Interface – End-User Level


13



Administration Level

14



Grid Database Access

15


Grid Database Access With OGSA-DAI

GDS gets a query via Perform Document

GDS Engine processspecified activities

GDS returns results


Grid Database Access With OGSA-DAI

16


Grid Data Mediation Service

Heterogeneities:Name in A is „Alexander Wöhrer“Name in C has to be combined

Distribution:3 data sources

Example Scenario


Grid Data Mediation Service - Architecture

17


OLAP (On-Line Analytical Processing)

Research Objectives: High- PerformanceGrid OLAP Services


Relational (ROLAP) vs. Multi- Dimensional Model (MOLAP)

Colour = {Red, Blue, White, Green}

Model = {Mini Van, Coupe, Sedan}

Two-Dimensional Model

18


ROLAP vs. MOLAP

SELECT Model, Year, Color, SUM(sales) AS SalesFROM SalesWHERE Model IN {´Ford´, ´Chevy´} AND

Year BETWEEN 1990 AND 1992GROUP BY CUBE (Model, Year, Color);

Model Year Color Sales

Chevy 1990 Blue 87Chevy 1990 Red 5Chevy 1990 ALL 92Chevy ALL Blue 87Chevy ALL Red 5Chevy ALL ALL 92Ford 1990 Blue 99Ford 1990 Green 64Ford 1990 ALL 163Ford 1991 Blue 7Ford 1991 Red 8Ford 1991 ALL 15Ford ALL Blue 106Ford ALL Green 64Ford ALL Red 8All 1990 Blue 186All 1990 Gree 64ALL 1991 Blue 7ALL 1991 Red 8Ford ALL ALL 178ALL 1990 ALL 255ALL 1991 ALL 15ALL ALL Blue 193ALL ALL Green 64

Model Year Color Sales

Chevy 1990 Red 5Chevy 1990 Blue 87Ford 1990 Green 64Ford 1990 Blue 99Ford 1991 Red 8Ford 1991 Blue 7

⇒


ROLAP vs. MOLAP

Cross Tab Data Cube

19


ROLAP vs. MOLAP

The ROLAP‘s summary records are storeddirectly int standard relational tables, withoutany need for data conversion. A complexanalytical query is cumbersome to express in SQL and it might not be efficient to execute.The array-based model, MOLAP (Multi-Dimensional OLAP), has the advantage thatnative arrays provide an immediate form of indexing for cube queries.


MOLAP

Each value of the attribute associated with thedimension determines a position of the dimension.The observed values, called measures are located at the intersections of the dimension positions. Theintersections are called cells and are populated withmeasures.Dimensional indexing: a unique integer value isassigned to each possible dimension position, and thecorresponding mapping information is stored in an appropriate index structure.

20


Requirements

Operation on large data sets

Centralized OLAP Service (parallel computing power can be included)

Distributed OLAP service

Federation of autonomous distributed OLAP services


Development Strategy

Network

OLAP Engine

OE

OE

OE

OE

21


Development Strategy (2)

Precondition: No open-source OLAP system availableDecision: development (in Java) fromscratchAdvantage: motivation for researchactivities addressing all facetsDisadvantage: a possible longimplementation curveFirst step: centralized sequential Grid OLAP service


Towards Centralized Service

GUIWorkflowEngine Mediator

OLAP

RD XMLD CSV

DSCL,OMML OMML XML

Data MiningEngine

PMML

PMML

22


Distributed OLAP – Aggregation of Computeand Storage Resources

Tuple Stream


OLAP Service

Virtual Cube

Sub Cube

Sub Cube

Slave 1

Slave 3

Master

Data

Sub Cube

Slave 2

Indexes

Index Service

query

answerXML

23


GridMiner Current Architecture

GUI

Knowledge Base

Service Configurators

Dynamic service composition engine (DSCE)

Data Base Access DM/OLAP services

Grid

Web

Use

r en

viro

nm

ent

DSCE Client


Towards an Open Service System

Clients(anywhere)

GUI (JWS)

Server(Vienna)

Cluster(Vienna)

Cluster(Linz)

Cluster(Innsbruck)

Workflow engine

WebApplications(Globus)

(Tomcat)

OLAP Master(Globus)

OLAP Slaves(Globus)

OGSA-DAI

Data mining services

(Tomcat)

(Globus)

24


Implementation/Technology

Globus 3.2

OGSA/DAI version 5

GUI – Workflow constructions/Results visualization (JGraph, Java Web Start, Java server pages)Service Configurators (Java server pages)

Workflow management – DSCE Client (OGSA)

Knowledge base – Configurations (XML,OWL)

Data mediation service (OGSA/DAI)


From Data Mining Grid to Semantic Grid

25


Towards the Next-Generation Web

The WWW is a milestone in the history of information sharing.A much more powerful infrastructure is neededto support e-Science.To overcome the Web‘s shortcomings, there isa vision of the next-generation Web, called theWisdom Web, also called Future Interconnection Environment. It is basedon the Web Intelligence concepts.


Topics Related to Web Intelligence

Diagram shows interrelated research topics from an intelligent Web-based, business perspective.

Ubuquitous computingand social intelligence

Web informationretrieval

Web miningand farming

Web informationmanagement

IntelligentWeb-based business

Knowledge networksand managementEmerging

Web technologyand Infrastructure

Intelligenthuman-Webinteraction

Web agents

Zhong, et al

26


Towards the Next-Generation Web (2)

Focusing on two approaches:

Improving the existing Web, whichincludes Semantic Web, Web Services, and Web IntelligenceEstablishing a new collaborativecomputing perspective of the Web –the Grid Intelligence


Grid Intelligence Support for the Wisdom Web

Wisdom Web

Web Intelligence

SocialIntelligence

GridIntelligence

. . .

. . . . . .

. . .

Next-Generation

Grids

Based on Cheung & Liu [2005]

27


Computational Grid

Data Grid

Data Minig Grid

Semantic Grid – 1st Generation

Current Grids

Next-GenerationGrid

Evolution ofthe Web

KnowledgeTechnologies

Evolution of HPCNMobileServices

Towards Next-Generation Grids


New EU IT Program

28


Data Mining Grids

VEGA (Italy) – an environment running on top of Globus 2. No public demonstration of VEGA is known.Discovery Net (UK, project finished in 2004) – based on Globus 2.GridMiner (Austria, active project) – supportfor all phases of the knowledge discoveryprocess and OLAP. Based on OGSA and OGSA-DAI.


Goals

The aim is not only knowledge discovery and sharing among scientists, but also

sustainable knowledge creation and scientific evolution

Reusing

Knowledge life cycle

Discovery

Sharing

Processing

29


Semantic Grids – 1st GenerationMost projects mainly focus on “Knowledgefor the Grid“ (using knowledge technologies to describe availability of Grid services, how they arediscovered, invoked, ...).

This approach is also partially followed in GridMinerOur new focus on “Knowledge on the Grid“ New framework planned: GridMiner+Research challenges:

Problem Solver Markup LanguagesAccess to and integration of knowledge basesHigh-performance distributed reasoningDynamic multi-level workflowsAutonomic features


Working with GridMiner+

GridMiner+Visible Grid

PartInvisible Grid

Part

Ontology, database, and Grid administrator User

administrationcommands &

queries

data explorationworkflow specifications,

results

problemspecifications,

solutions

30


GridMiner+ - Architecture

Auto

nom

ic S

upport

Intelligent interface

Knowledge management infrastructure

Knowledge Provider

Problem Solver

Generic Grid Services

Globus Toolkit

Data mining infrastructure

GridMiner


Knowledge Base

Metadata

Rules

Facts

Ontologies

Models

InferenceEngine

Rules and FactsGenerator

31


Transform request queries

Validate requestwith ontologies

Process request

User Problem solver Knowledge provider(GridMiner)

Search KBs

Processresults

Transform request to KDD task

KDDWorkflow

Generate rules

Processresults

Find data miningactivity

Apply rules on domain ontology

Generate outputs

KDD

Problem Solving WorkflowsStatic and Dynamic Features

understandable?

yes

no

Demand

Response

Response

PSML – Problem Solver Markup Language

DB AccessWorkflow


PSMLs

SWRL Semantic Web Rule Language (W3C): combination of OWL with RuleMLβ-PSML (Su et al.): combination of OWL withHorn clauses; it can represent multi-argumentrelations.Query example (Prolog notation):

?- hasHypertensionRisk(patient#12756)

hasHypertensionRisk, its values, etc. aredefined in the appropriate domain ontology

32


Rules

The rulehasHypertensionRisk(patient) :-

systolicGreaterThan(patient,140),diastolicGreaterThan(patient,90),periodInYearsGreaterThan(patient,2).

If the rule is not in the KBs, the systemstarts data mining and tries to derive therule automatically.


Distributed Reasoning

Needed for solving problem in a large scale Web and Grid environment.Appropriate PSML features need to be investigated.Two architectures can be considered:

KB KB KB

IE

GN GN

GN

GN

GN – Grid NodeKB – Knowledge BaseIE – Inference Engine

IE KB

IE KB IE KB

GN

GN

GN

33


Autonomic Computing Support

Providing data mining and knowledgemanagement services transparently

Relieving professionals from activities thatare of a system nature

Allowing them to concentrate their effortson their area activities

Offering computing and communivationservices quickly, reliably and securely.


Conclusions

In the near future, the development of intelligent applications will be based on intelligent interconnection environments (IIEs)

IIEs will be supported by the Web and Grid Intelligence

Grid Intelligencerealized by Semantic Gridsimportant components

Data Mining Gridsknowledge basesefficient reasoning mechanisms

34


Demo Presentation (Video) Follows

- from a vision to design and implementation peter...

Documents