© geodise project, university of southampton, 2001-2005. geodise & geodiselab simon cox...

© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/

Geodise & GeodiseLAB

Simon Cox

University of Southampton

21st April 2005

Grid Enabled Optimisation and Design Search for Engineering

(GEODISE)Southampton, Oxford and ManchesterSimon Cox- Grid/ W3C Technologies

and High Performance ComputingGlobal Grid Forum Apps Working Group

Andy Keane- Director of Rolls Royce/ BAE Systems University Technology Partnership in Design Search and Optimisation

Mike Giles- Director of Rolls Royce University Technology Centre for Computational Fluid Dynamics

Carole Goble- Ontologies and DARPA Agent Markup Language (DAML) / Ontology Inference Language (OIL)

Nigel Shadbolt- Director of Advanced Knowledge Technologies (AKT) IRC

BAE SYSTEMS- Engineering

Rolls-Royce- Engineering

Fluent- Computational Fluid Dynamics

Microsoft- Software/ Web Services

Intel- Hardware

Compusys- Systems Integration

Epistemics- Knowledge Technologies

Condor- Grid Middleware

© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/

The GEODISE Team ...

• Richard Boardman• Sergio Campobasso• Liming Chen• Mike Chrystall• Trevor Cooper-Chadwick• Simon Cox• Mihai Duta• Clive Emberey• Hakki Eres• Matt Fairman• Mike Giles• Carole Goble• Ian Hartney• Tracey Hunt• Zhuoan Jiao

• Andy Keane• Marc Molinari• Graeme Pound• Colin Puleston• Nicola Reader• Angus Roberts• Mark Scott• Nigel Shadbolt• Wenbin Song• Paul Smart• Barry Tao• Lisa Telfer-Brunton• Jasmin Wason• Fenglian Xu• Gang “Luke” Xue


Distributed Systems 2005

Network

IP

HTTP

(HTML)

Compute/ Data

Moore’s Law

Future ServicesFlexible, Seamless, and Secure Sharing

Software

(HTML) XML

Web Services

ASCII


Negative Scarf InletConventional Inlet

0 1 2

Applications - GeodiseCFD 3D engine nacelle optimisation

Goal - reduce ground noise generated by fan when plane takes off.

Optimise aerodynamic performance when scarf angle is varied.

Problem definition

Design of Experiment

Response surface modelling

Optimisation on Response surface

Validation

ProEngineer CAD(Condor Pool)

Gambit Meshing(Globus Compute)

Fluent CFD(Globus Compute)

Workflow for aerodynamic shape optimisation using CAD, Gambit, and Fluent

© Geodise Project, University of Southampton, 2001- 2004.http://www.geodise.org/

Applications – GEM

Electromagnetic optimisation

• Large number of designs, parameters and solutions

• Query for a particular data range to post-process

Transmission of light properties through a photonic crystal

Parameters are radius of holes and light wavelength

APPLICATION SERVICE

PROVIDERCOMPUTATION

GEODISE PORTAL

OPTIMISATION

Engineer

Parallel machinesClusters

Internet Resource ProvidersPay-per-use

Optimisation archive

Intelligent Application Manager

Intelligent Resource Provider

Licenses and code

Session database

Design archive

OPTIONSSystem

Knowledge repository

Traceability

Visualization

Globus, Condor, OGSA

Ontology for Engineering,

Computation, &Optimisation and Design Search

CAD SystemCADDSIDEASProE

CATIA, ICAD

AnalysisCFDFEMCEM

ReliabilitySecurity

QoS

Geodise will provide grid-based seamless access to an intelligent knowledge repository, a state-of-the-art collection of optimisation and search tools,

industrial strength analysis codes, and distributed computing & data resources

GEODISE


Grid-Enabled Scripting Environment

• Why use scripting languages?– Flexibility – High-level functionality– Quick application development– Extend the user’s existing PSE– … Acts as execution/ enactment engine too

• Approach– Matlab initially chosen as the hosting environment– Now working with Python/ Jython

MatlabPython/ & /Jython

Hardware

Management

Grid

Enterprise

Applications

Collaboration

Coordination

Hardware

Management

Grid

Enterprise

Applications

Collaboration

Coordination

Engineering Applications of the Grid via Geodise

Problem Solving Environment

Bringing underlying infrastructure into the Problem

Solving Environment

“Useful into useable”

Grid

Hardware

Matlab, Python, etc

“Heartbeat”Enterprise

“Dial-tone”

Geodise Toolkits

Geodise deliverables

• Grid Compute toolbox Cycle stealing High-performance computing Integration with heterogeneous environments

(Windows, Linux, Solaris, …)• Grid Data toolbox

Raw files into journaling filesystem Metadata in relational database (Oracle, DB2, SQL

Server, mySQL…) XML processing capability

• Intelligent Workflow composition, execution and monitoring tool Rule and case-based composition of ‘best’

workflow (semantics and knowledge driven) Deployment to compute/ data resources System-level “heartbeat” User/ Application level “dial-tone” (security,

resource availability, network weather, …)

Problem Solving Environments e.g. Matlab,

iSight Python, ModelCenter,…

Grid-enabled Toolboxes for Engineers

Middleware (Grid)

Hardware

Opti

mis

ati

on

`Tools

Designed around web-services to inherit and leverage

Technologies (e.g. security, reliability) and Development tools


Geodise Tools

• A number of Grid-enabled design tools have been developed– Compute Toolbox: certificate based authentication, job

submission, and file transfer, etc.– Database Toolbox: archive files and data, custom metadata,

locate and retrieve data, etc.– XML Toolbox: convert Matlab structures to and from XML– Workflow Editor: graphical problem solving environment which

allows users to construct and configure workflows for the Grid– OptionsMatlab: Matlab interface to the Options design

exploration system– Knowledge Toolbox: Intelligent Design advice

Hardware

Management

Grid

Enterprise

Applications

Collaboration

Coordination

Hardware

Management

Grid

Enterprise

Applications

Collaboration

Coordination

Geodise Database

Globus Server

GraphicalWorkflow Tool

Matlab Scripting Environment

Compute ToolboxDatabase Toolbox

OptionsMatlab

Compute ToolboxAPI

Database ToolboxAPI

Knowledge ToolboxAPI

KnowledgeRepository Metadata

andStructure

Repository

Computation

Globus Server #1

Globus Server #N

Condor Cluster

Globus Server /Condor Node

Condor Master

Condor Node #1

Engineer

Knowledge Toolbox

Condor Node #N

Applications

Pro/Engineer

ICAD

Catia

IDEAS

CAD

Gambit/Fluent

Abaqus

Catia

HYDRA

Analysis

Globus Servers

Glob

us and

Con

dor

Job M

ana

gers

Java Matlab

MetadataWebservice

Physical files,scripts, and

outputs

Results by GridFTP

ResultsResults

LocalFilesystem

Matlab Script

Grid JobSubmission

Results

Metadata by webservice

Geodise Architecture

XML Toolbox


Sponsors


GeodiseLab Activities for the OMII

• Harden & deliver Geodise toolboxes– Geodise Compute Toolbox

– Geodise Database Toolbox

– XML Toolbox for Matlab

• Documentation• Testing • Extend existing functionality• Train and support users• ... GGF: Simple API for Grid Applications (SAGA)

Working Group


Compute Toolbox: Aims

• Support access to whichever compute resources users wish to exploit– Globus GT2– OMII services– Condor

• Provide access from alternative environments– Matlab– Jython– cPython

• Leverage existing APIs wherever possible


Compute Toolbox: Hit-list

Matlab

Geodise Java API

Jython cPython

CondorNative

?

.py files .py files

WindowsHPC/Grid

CondorWeb Service

GlobusGT2

OMIIServices

Java Client Java CoG OMII API.NET Client

.m files


GeodiseLab Compute Toolbox

Available Now


Compute Toolbox Release

Matlab

GlobusGT2

OMIIServices

Geodise Java API

Jython cPython

CondorNative

CondorWeb Service

Java CoG OMII APIJava Client

?

.py files .py files.m files


Toolbox Architecture

Matlab Environment

Globus GT2 Resource GRAM

ServiceGridFTPService

gd_jobsubmit.m

Java CoG

gd_getfile.m

script.m

GridFTP file transfer

GRAMjob submission


Certificate Management Functions

gd_certinfo Returns information about the user's certificate

gd_createproxy Creates a Globus proxy certificate

gd_proxyinfo Returns information about the user's proxy certificate

gd_proxyquery Queries whether a valid proxy certificate exists

gd_destroyproxy Destroys the local copy of the user's Globus proxy certificate


Certificate Management

C:\GEODISE

>> gd_createproxy


Certificate Management

C:\GEODISE

>> subject = gd_certinfosubject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme poundissuer : C=UK,O=eScience,OU=Authority,CN=CA,[email protected] date : Wed Sep 22 16:31:24 BST 2004end date : Thu Sep 22 16:31:24 BST 2005 subject = /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=graeme pound >> isvalid = gd_proxyinfosubject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound,CN=proxyissuer : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme poundtype : full legacy globus proxystrength : 512 bitstimeleft : 11 h, 59 min, 30 sec

isvalid =

1

>>


Job Submission Functions

gd_jobstatus Gets the status of a Globus GRAM job

gd_jobsubmit Submits a compute job to a Globus GRAM job manager

gd_jobpoll Queries the status of a Globus GRAM job until complete

gd_jobkill Kills a Globus GRAM specified by a job handle

gd_chmod Changes file permissions of a file on a Globus resource

gd_condorsubmit Submits a job to a Condor pool via a Globus resource

gd_submitunique Submits a GRAM job to a unique working directory


Job Submission

C:\GEODISE

>> RSL = '&(executable="/bin/date")(stdout="date.out")';>> host = 'grid-compute.oesc.ox.ac.uk';>> jobhandle = gd_jobsubmit(RSL, host)

jobhandle =

https://grid-compute.oesc.ox.ac.uk:30001/30705/1098694366/

>> isdone = gd_jobpoll(jobhandle)

isdone =

1

>>


GridFTP File Transfer

gd_getfile Retrieves a remote file using GridFTP

gd_putfile Puts a remote file using GridFTP

gd_transferfile Performs a third-party file transfer using GridFTP

gd_makedir Creates a remote directory using GridFTP

gd_listdir Lists the contents of a directory on a GridFTP resource

gd_fileexists Tests the existence of files on a Globus resource

gd_rmdir Deletes a remote directory using GridFTP

gd_rmfile Deletes a remote file using GridFTP

gd_rmuniquedir Deletes a remote directory and its contents


File Transfer

C:\GEODISE

>> exists = gd_fileexists(host, 'date.out')

exists =

1

>> gd_getfile(host,'date.out','localfile.txt');>> type('localfile.txt')

Mon Oct 25 09:52:46 BST 2004

>> gd_rmfile(host, 'date.out');>> exists = gd_fileexists(host, 'date.out')

exists =

0

>>


Resource Testing

gd_servermetrics Performs tests upon a list of Globus resources

gd_testauthentication Tests authentication with a Globus resource

gd_testfiletransfer Tests file transfer to a Globus resource

gd_testjobsubmission Tests the job submission to a Globus resource

gd_timeauthentication Times authentication to a Globus resource

gd_timefiletransfer Times file transfer to a Globus resource

gd_timejobsubmission Times a job submission to a Globus resource


Resource Testing

C:\GEODISE

>> servers(1).name = 'grid-compute.leeds.ac.uk';>> servers(2).name = 'escience-dept2.sesnet.soton.ac.uk';>> results = gd_servermetrics(servers);>> disp(results(1))

name: 'grid-compute.leeds.ac.uk' authentication: 188 jobsubmission: 672 filetransfer: 984

>> disp(results(2)) name: 'escience-dept2.sesnet.soton.ac.uk' authentication: 125 jobsubmission: 485 filetransfer: 4938

>>


GeodiseLab XML Toolbox

Available Now


Engineer’s familiar scripting environment

3C0003

2B0002

1A0001

ValueNameID

XML Toolbox

Web Technology & Services

Databases

Access

Communication


Features

• Powerful Matlab to XML interface• Allows transparent conversion between proprietary Matlab

data types and standard XML• Brings flexibility of open XML Standard to Matlab• Easy-to-use interface (4 simple core functions:

xml_save, xml_load, xml_format, xml_parse)• Extensive documentation & direct help functionality• Large range of potential applications:

– Science, Engineering, IT, Web, …

Toolbox

Toolbox

<struct xmlns="http://www.geodise.org/matlab.xsd" idx="0" fields="name id param"> <char idx="1" name="name" size="1 14">eng_opt_design</char> <char idx="1" name="id" size="1 11">1022223-779</char> <struct idx="1" name="param" size="1 1" fields="width height material"> <double idx="1" name="width" size="1 1"> 100 </double> <double idx="1" name="height" size="1 1"> 87.9 </double> <double idx="1" name="material" size="1 1"> 0.07244 </double> </struct></struct>

<root> <name>eng_opt_design</name> <id>1022223-779</id> <param> <width> 100 </width> <height> 87.9 </height> <material> 0.07244 </material> </param></root>

DBFilestoreStandard File Formats

(eg. STEP-XML, …)


XML Toolbox in action

• Jan 2004 v2.0 released• Nov 2004

– Downloads from soton.ac.uk/~gridem: 227– Downloads from mathworks.com/matlabcentral: 882

• Apr 2005 v3.0– > 1500 downloads total

• Large user base– Academic & Commercial– Extensive testing & feedback community– Market pull

• So far very positive feedback from users• Matlab Central user rating: 5 / 5• Linked by Mathworks help for XML in Matlab


Aalesund College: Read data from databaseAberdeen University: Export analysis results from Matlab to XMLADVA: to read existing XML-files in a Matlab structure and do some calculationsAdvanced Tech Center: configuring xml input into matlabAlfred-Wegener-Institut: Handle data from echo sounderAMAT: Evaluation of XML input in matlabAnna University: We are programmed to develop a new tool for analysing discrete event system using matlab.Applied Math Labolatory, Ecole Centrale ParisAston University: To read date from xml files in matlabAvantium Technologies BV: Transferring HTS data stored as XML into MatlabBAE Systems: Data import/export between MATLAB and Delphi; Generation of XML files to control a BAE SYSTEMS Software Defined RadioBeb CorpBosch GmbHCalypso Medical: looking at xml files in matlabCarnegie Mellon university: Use XML in our facial expression analysis projectCenter for Advanced Spatial Technologies: Writing least squares adjustments for geodetic observations as part of research in new techniques for parameter estimation combining photogrammetric and terrestrial observations. Data is varied but structured and I am interested in using XMLChouaib Doukkali Universit: Numericals simulations of the physics phenomenonsChristian-Albrechts-Universitat zu Kiel (CAU): http://busselab.uni-kiel.deCity College of CUNY: Projects using Matlab and XML technology.CMS: For trialCold Spring Harbor Laboratory: Storing and parsing neuronal dataConestoga college: intro to telecommeCrystal EngineeringCU: Sending sending commands to matlab via xml, outputting results of calculations from matlab in xml to be used by other products

Daimler Chrysler: test

DeAM: xml reading

DECO

DEIMOS Engenharia: Reading/writing configuration files

DGA Defence, France: For managing data files used in simulation's input

DSTO: input parameters to radar model

EADS CRC: InformationEmyo: For teachingENI: Reading & Modifying XML documentsEPFL: Evaluation for data exchange between matlab and other softwareFairchild Dornier: Currently I make a research, how I can efficiently store my matlab data structure in a text editable data format, that's why I'am looking for a xmlread and a xmlwrite function.Financial EnginesFIOTFraunhofer Germany: The last aim of our research is to provide user a platform for maintenance oriented decision via web service.

38 sample institutions


GeodiseLab Database Toolbox

Available Now


Database Toolbox – Aims

• Make data easier to manage, find and share• Store data with descriptive information

– Standard and user defined metadata– Query over metadata to easily locate data

• Familiar interface for engineers– Wrap toolbox as Matlab/Jython functions– Easy integration into user scripts


Database Toolbox – Features

• Archive files and data structures with metadata

• Group related data• Simple query interface• Retrieve data based on ID rather than

location• Security

– User grants permission for others to access their data– Certificate-based authentication and authorisation– Secure Web service access to central database

© Geodise Project, University of Southampton, 2005.http://www.geodise.org/

Functions Overview

gd_archive Stores a file or structure with metadata into an archive

gd_query Queries over metadata or structures in the archive

gd_retrieve Retrieves a file or structure from the archive to the local machine

gd_addusers Grants users permission to access some data

gd_datagroup Creates a new datagroup to aggregate files and structures

gd_datagroupadd Adds a file or structure to a datagroup

gd_display Displays the results of a query


GeodiseLab Database Toolbox Worked Example

GridClient

QueryService

Globus ServerXML Toolbox Java CoG

LocationService

AuthorisationService

MetadataService Database

Generate Data

GridClient

QueryService


LocationService



Metadata

GridClient

QueryService


LocationService



Archive

XML Toolbox Java CoGGlobus Server

LocationService



GridClient

QueryService


LocationService



Generate More Data

Java CoGGlobus Server

XML Toolbox

LocationService



GridClient

QueryService


LocationService



Group Files

XML Toolbox



GridClient

QueryService


LocationService



Query

XML Toolbox

QueryService


Database

GridClient

QueryService


LocationService



Retrieve

Java CoGGlobus Server

LocationService


Database


Applications - GENIE

GENIE supports environmental scientists modelling long term climate change.

Optimising a parameterised model over a multi-dimensional state space.

Database used for monitoring, sharing, post-processing and fault tolerance.

Earth system model tuning studies


Distributed Systems: 2005 and beyond?

Network

IP

HTTP

(HTML)

Compute/ Data

Moore’s Law

Software

(HTML) XML

Web Services

ASCII

Experiences

Services

RobustnessSecurityReliabilityVisualizationQuality of ServiceKnowledge TechnologiesCollaboration tools & frameworks

Platform StrategiesDevelopment Tools

Systems ArchitectureDistributed Data ManagementProblem Solving Environments

Algorithms, Methods and LibrariesHeterogeneous Distributed

Computing

© geodise project, university of southampton, 2001-2005. geodise & geodiselab simon cox...

Documents

aerodynamic shape optimisation

royce engineeringfluent

search tools

aerodynamic performance

groupandy keane director

particular data range

engineering geodisesouthampton

seamless access