ogsa-dai: patterns and functions supercomputing ‘06 aist stand malcolm atkinson director e-science...
TRANSCRIPT
O G S A -D A I
OGSA-DAI: Patterns and Functions
Supercomputing ‘06AIST Stand
Malcolm Atkinson
Director e-Science Institute & e-Science Envoy
www.nesc.ac.uk14th November 2006
Outline of Talk Reminder about OGSA-DAI
Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team
The new architecture Motivation Features Status
The Road ahead Engineering Challenges Research Challenges
OGSA-DAI OGSA-DAI provides:
WS framework for wrapping data resources Extensible:
Functionality can be added Client library Move computation to be close to the data
Have a workflow like language Encapsulate multiple interactions in one-interaction
Works with multiple Grid Fabrics OMII-UK container, GT4.*, Tomcat + Axis Soon (OMII-Europe project): Unicore, gLite 3.0
Simple DB operations SQL
Data Resource Accessor
Relational
XMLDBData
Resource Accessor
Data Service
Resource
Data Service
Resource
Data
Service
Data
Service
XPath or XQuery
5
O G S A -D A IExposing Databases
Data Resource Accessor
Relational
XMLDBData
Resource Accessor
Data Resource Accessor
Data Service
Resource
Files
Data Service
Resource
Data Service
Resource
SQLOne
XMLOne
FilesOne
Data
Service
Relational Multi-resourcesMultiple Data Resource
AccessorMulti
Resource
Data Service
Resource
MultiOne
Data
Service
Data
Service
Relational
SQL Query
Multi Results
SQL Query
Results x M
Data Service
Resource
Data
Service
Relational
SQL Query SQL QueryResults Results
SQL QuerySQL Query ResultsResults
SQL Query SQL QueryResults Results
Delegate DAI task completion
consumer
Data
Res
ourc
e
DR messages
Client OGSA-DAIRequest & Response D
ata
Res
ourc
e
DR messages
Data
del
iver
y
OGSA-DAI
Request & Response
Multi-ServiceIntegratione.g. DQP
Client
DR1
DR2
DR3
Data
Res
ourc
e
OGSA-DAI
Data
Res
ourc
eD
ata
Res
ourc
e
DR mes
sage
s
DR messages
DR messages
Data
Res
ourc
e
OGSA-DAI
Data
Res
ourc
eD
ata
Res
ourc
e
DR mes
sage
s
DR messages
DR messages
Data
Res
ourc
e
OGSA-DAI
Data
Res
ourc
eD
ata
Res
ourc
e
DR mes
sage
s
DR messages
DR messages
Request, R
esponse & D
ata Transport
Req
uest
, Res
pons
e & D
ata
Tra
nspo
rt
Request & Response
9
O G S A -D A IClients & Authorisation
Data Resource Accessor
Data
Service
Data Service
ResourceClient
Perform Document
SQLOne
Relational
Perform Document
SQL Query
ResultSet
SQL Query
ResultsResponse Document
Response Document
Authorization points
Used by a number of projects
OGSA-DAI
More info from: http://www.ogsadai.org.uk/about/projects.php
International Cooperation and Recognition
USA:o Globus Allianceo IBM Corporationo caBIGo BIRNo Indiana University o GridSphereo GEONo LEADo MCSo NCSAo Secure Data Grido UNC
Japan:o AISTo BioGrido NAREGI
Europe:o CERNo DataMiningGrido GridMinero GridSphereo inteligrido N2Grido OntoGrido Provenanceo SIMDAT
UK:o OMIIo NGSo NCeSSo NIeeSo AstroGrido BioSimGrido BRIDGESo CancerGrido ConvertGrido eDiaMonDo EDINAo First Group plco Fujitsu Labs Europeo GEDDMo GeneGrido Genomic Technology and Informaticso GOLDo Human Genetics Unito IBM UKo myGrido Oracle UK
China:o CASo ChinaGrido cnGrido INWA
Australia:o Curtin Business Schoolo INWA
TutorialsBoston CambridgeCERN ChicagoEdinburgh LondonSan Francisco SeattleSeoul SingaporeTokyo ISSGC 03 to 06
DIALOGUE workshopsColumbus, Edinburgh, Indiana, Vienna
Chicago, Manchester, San Diego
South Korea:o KISTI
China40%
United Kingdom15%
United States11%
Germany3%
Japan5%
Italy2%
France3%
Austria1%
Others20%
1485 registered users5250+ downloads
Outline of Talk Reminder about OGSA-DAI
Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team
The new architecture Motivation Features Status
The Road ahead Engineering Challenges Research Challenges
Features of Future OGSA-DAI Comprehensive library of Integration Activities
Putting the I in OGSA-DAI You will still add your own
Perhaps less often With more examples to start from With a new OGSA-DAI Engine for support With new Activity base classes
Connected by “pipes” carrying Streams of values – scalability is a major motivator
Each with own threads Exploiting concurrent processing Possibly in different VMs – resilience a major motivator
Support for Iteration Expect Request documents
To contain more sophisticated DAGs Branching and merging activities Synchronous and dynamic data consumption
Auto-iteration over (Lists of) values Iteration activities
Manage the iteration of a controlled graph of activities
17
O G S A -D A IMulti-resource requests
Data Request
Execution Service
Data Request Execution Resource
Client
Data Resource Data
Data Resource Data
Data Resource Data
MyDRER
One
Two
Three
Perform Document
myQuery(One) myQuery(Two)
myIntegrateData
deliverGridFTP
Client-server interaction
Service-resource association
Resource-resource association
Resource-database association
18
O G S A -D A IAn expanded resource model
Data Request
Execution Service
Data Request Execution Resource
Client
Data Resource
Information Service
Data Resource
Session Management
Service
Session
Data
Data Resource Data
Data Resource Data
SessionSession
SessionSessionRequest
Request Management
Service
MySession123
MyDRER
One
Two
Three
MyRequest123456
19
O G S A -D A IPersistence
OGSA-DAI State
Data Request
Execution Service
Data Request Execution ResourceClient
Data Resource
Information Service
Data Resource
Session Management
Service
Session
Data
SessionSession
SessionSessionRequestRequest
Management Service
OGSA-DAI Server
Another OGSA-DAI Server
20
O G S A -D A IPersistence
Caching Reduce overheads of OGSA-DAI-database communications
Allows configuration and state to persist between container shutdowns and restarts
Sticky resources e.g. data sources and sinks In-memory – only available via server that created them
API File-based implementation for backwards compatibility Relational implementation for compatibility with common
OGSA-DAI databases Extensibility point
21
O G S A -D A IScalability and robustness
OGSA-DAI State
Client
OGSA-DAI cluster
Front-end Router
Monitor & Restart
Warning: not in next release – ODE contains the foundations
22
O G S A -D A IScalability and robustness
Release 3 will afford future provision of clustering and load balancing Clustered OGSA-DAI servers
Different JVM Different machine
Load balancer Simple e.g. round robin Complex e.g. decide on basis of current load of each server
Scalable Increases number of requests that OGSA-DAI can handle
concurrently Robust
Seamless operation in the face of errors e.g. server failures Sticky sessions and resources
Session and resource state may be non-serializable or be lightweight in-memory components
Need to dispatch resources to their “owner” JVM
23
O G S A -D A IOther planned features
Support for registries Guidelines on how to best use standard registries
(e.g. GRIMOIRES or GT4 ServiceGroups) and tools to help you do so within OGSA-DAI
Improved support for message-level and transport-level security including delegation
Concurrent or closely-following releases of A new implementation of DAIS
OGSA-DAIR and OGSA-DAIX A re-factored OGSA-DQP with a Java-compliant query
compiler and which runs under Linux, Solaris or Windows
Support for generic activities Powerful general purpose activities
Parameterised by Java methods Any data transformation or integration Task for application partitioned to writing simple methods Iteration and data management provided
ExamplesTupleGenericSort TupleGenericProductSelectProject
TupleGenericSelectProject TupleGenericUnion
TupleGenericMultiSelectProject TupleGenericIntersection
TupleGenericJoinSelectProject TupleGenericDifference
TupleGenericGroupBy
Outline of Talk Reminder about OGSA-DAI
Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team
The new architecture Motivation Features Status
The Road ahead Engineering Challenges Research Challenges
Engineering Challenges Integrating / supporting all activities
Choices of internal data formats OMII-UK & OGSA-DAI code Contributors code
Robust & scalable ODE implementations Continuous operation
deployment, configuration, security & management Supporting but containing users’ code
Integration with other data services Replica & cashing services Grid file services Storage management services
Research Challenges High-level notations
Auto-Optimisation, Partitioning & Parallelisation Meta-data driven DAI
Automatic generation of DAI workflow detail Visual Comprehension
Visual forms of DAI workflows Visualisation of enactment progress Provision of control and diagnosis aids
Relationship with other workflows
28
O G S A -D A IFurther information The OGSA-DAI project site
http://www.ogsadai.org.uk
The DAIS-WG site http://forge.gridforum.org/projects/dais-wg
Recent talkshttp://www.ogsadai.org.uk/documentation/presentations/GW2006GGF18/
Design in Progress
http://www.ogsadai.org.uk/documentation/Design_documents/ OGSA-DAI users mailing list
[email protected] General discussion on OGSA-DAI, data and the grid
Formal support for OGSA-DAI releases http://www.ogsadai.org.uk/support [email protected]
OGSA-DAI training courses