The Grid, Grid Services and the Semantic Web: Technologies and Opportunities
Dr. Carl KesselmanDirectorCenter for Grid TechnologiesInformation Sciences InstituteUniversity of Southern California
Outline
What are Grids? Grid technology
- Globus and the Open Grid Services Architecture
Grids and the Semantic Web
How do we solve problems? Communities committed to common goals
- Virtual organizations
Teams with heterogeneous members & capabilities
Distributed geographically and politically- No location/organization possesses all required skills
and resources
Adapt as a function of the situation- Adjust membership, reallocate responsibilities,
renegotiate resources
The Grid Vision
“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”- On-demand, ubiquitous access to computing, data,
and services
- New capabilities constructed dynamically and transparently from distributed services
“When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances”
(George Gilder)
Biomedical InformaticsResearch Network (BIRN)
Evolving reference set of brains provides essential data for developing therapies for neurological disorders (multiple sclerosis, Alzheimer’s, etc.).
Today - One lab, small patient base- 4 TB collection
Tomorrow- 10s of collaborating labs- Larger population sample- 400 TB data collection: more
brains, higher resolution- Multiple scale data integration and
analysis
National Virtual Observatory
Xray (ROSAT) theme
Change scale
Change theme
http://virtualsky.org/fromCaltech CACRCaltech AstronomyMicrosoft Research
Optical (DPOSS)
Coma cluster
Virtual Sky has140,000,000 tiles
140 Gbyte
Living in an Exponential World(1) Computing & Sensors
Moore’s Law: transistor count doubles each 18 months
Magnetohydro-dynamics
star formation
Living in an Exponential World:(2) Storage
Storage density doubles every 12 months Dramatic growth in online data (1 petabyte =
1000 terabyte = 1,000,000 gigabyte)- 2000 ~0.5 petabyte
- 2005 ~10 petabytes
- 2010 ~100 petabytes
- 2015 ~1000 petabytes?
Transforming entire disciplines in physical and, increasingly, biological sciences; humanities next?
An Exponential World: (3) Networks(Or, Coefficients Matter …)
Network vs. computer performance- Computer speed doubles every 18 months
- Network speed doubles every 9 months
- Difference = order of magnitude per 5 years
1986 to 2000- Computers: x 500
- Networks: x 340,000
2001 to 2010- Computers: x 60
- Networks: x 4000
Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.
The Grid World: Current Status
Dozens of major Grid projects in scientific & technical computing/research & education
Considerable consensus on key concepts and technologies- Open source Globus Toolkit™ a de facto standard for
major protocols & services- Far from complete or perfect, but out there, evolving
rapidly, and large tool/user base Industrial interest emerging rapidly Opportunity: convergence of eScience and
eBusiness requirements & technologies
The Next Step Globus leverages standard protocols
- TLS, LDAP, X.509, HTTP
- Only TCP in common
Is there a better foundation for Grid functions- More unified protocol stack (common base)
- Better support for virtualization
- Leverage commodity infrastructure
“Web Services” Increasingly popular standards-based framework for
accessing network applications- W3C standardization; Microsoft, IBM, Sun, others
WSDL: Web Services Description Language- Interface Definition Language for Web services
SOAP: Simple Object Access Protocol- XML-based RPC protocol; common WSDL target
WS-Inspection- Conventions for locating service descriptions
UDDI: Universal Desc., Discovery, & Integration - Directory for Web services
Transient Service Instances
“Web services” address discovery & invocation of persistent services- Interface to persistent state of entire enterprise
In Grids, must also support transient service instances, created/destroyed dynamically- Interfaces to the states of distributed activities
- E.g. workflow, video conf., dist. data analysis
Significant implications for how services are managed, named, discovered, and used- In fact, much of our work is concerned with the
management of service instances
OGSA Design Principles
Service orientation to virtualize resources- Everything is a service
From Web services- Standard interface definition mechanisms: multiple
protocol bindings, local/remote transparency
From Grids- Service semantics, reliability and security models
- Lifecycle management, discovery, other services
Multiple “hosting environments”- C, J2EE, .NET, …
The Grid Service =Interfaces + Service Data
Servicedata
element
Servicedata
element
Servicedata
element
GridService … other interfaces …
Implementation
Service data accessExplicit destructionSoft-state lifetime
NotificationAuthorizationService creationService registryManageabilityConcurrency
Reliable invocationAuthentication
Hosting environment/runtime(“C”, J2EE, .NET, …)
Given a set of Services?
How do we do a better job of finding out what services we want to use
How do we do a better job of configuring services
How do we do a better job of composing and nesting services
Answer: Do a better job of representing services
Deeper representation of services
Information is captured via structure- X.509 certificates, MDS models, CIM schema,
Metadata
Knowledge expresses relationships between entities- Concepts and relationships
- Logical framework to inference over relationships
Vision
“The Semantic Web is an extension of the current Web in which information is given a well-defined meaning, better enabling computers and people to work in cooperation. It is the idea of having data on the Web defined and linked in a way that it can be used for more effective discovery, automation, integration and reuse across various applications. The Web can reach its full potential if it becomes a place where data can be processed by automated tools as well as people”
From the W3C Semantic Web Activity statement
Resource Description Framework
Ontologies Everywhere
What happens if knowledge permeates the Grid- Data elements
- Service descriptions (service data elements)
- Protocols (e.g. policy, provisioning)
More dynamic and general model then Semantic Web- OGSA lifetime model
- OGSA SDE model
Cognative Grid
Grid Services + Ontologies + Knowledge Driven Services
Examples- Knowledge driven matchmaking
- Agent based service composition
- High-level planning and resource discovery
- Knowledge based provisioning
Some people are using term “semantic grid” to discribe Grid Services+Knowlege
SCEC Modeling Environment
Knowledge Base
OntologiesCurated taxonomies,
Relations & constraints
Pathway ModelsPathway templates,
Models of simulation codes
CodeRepositories
Data & SimulationProducts
Data Collections
FSM
RDM
AWM
SRM
Storage
GRIDPathway Execution
Policy, Data ingest, Repository access
Grid ServicesCompute & storage management, Security
DIGITALLIBRARIES
Navigation &Queries
Versioning,Topic maps
MediatedCollectionsFederated
access
KNOWLEDGEACQUISITION
Acquisition InterfacesDialog planning,
Pathway constructionstrategies
Pathway AssemblyTemplate instantiation,
Resource selection,Constraint checking
KNOWLEDGE REPRESENTATION & REASONINGKnowledge Server
Knowledge base access, InferenceTranslation Services
Syntactic & semantic translation
Pathway Instantiations
Computing
Users
DOCKER: Publishing SHA Code
SCEContologies
AS97
msg
types
AS97 ontology
constrs
docs
User specifies: Types of model parameters Format of input messages Documentation Constraints
User Interface
ConstraintAcquisition
ModelSpecification
DOCKER
Web Browser
WrapperGeneration
(WSDL, PWL)
AS97
(Y. Gil, USC/ISI)
Recommends other models
Yes
Did you know that [Sadigh97] is a good model for dist >80 miles?
Automatically Generates Interface
Automatically Generates KR Description
myGrid Project - bioinformatics
Imminent ‘deluge’ of genomics data
- Highly heterogeneous, Highly complex and inter-related
Convergence of data and literature archives
1. Database access from the Grid
2. Process enactment on the Grid
3. Personalisation services4. Metadata services
Grid Services + Ontologies
Carol Gobel, U. Manchester
Resource selection: Matchmaking
Providers and requesters describe themselves- Synactic description
> Structured or Semi-structured
A Matchmaker matches compatible classads- Match based on attribute name, simple prioritization
Semantic matchmaking- Inference based matching (e.g. CIM+relations)
- Automatic classification (e.g. description logic)
- Leverage domain specific ontologies
Pegasus: Planning for Execution in Grids
Create workflow to create virtual data- Domain specific and
generic rules
Map Workflow unto Grid resources- System state via Grid
services (MDS, RLS,…)
- Global and local optimization criteria Condor-G/
DAGMan
TransformationCatalog
RLS
(1) Abstract Workflow(DAG)
(3) Logical File Names(LFNs)
(4) Physical File Names(PFNs)
Chimera
Request Manager
(18) Results
VDL GeneratorSubmit File
Generator forCondor-G
Concrete PlannerAbstract and
Concrete Planner
(9) Concrete DAG
(10) ConcreteDAG
(11) DAGMan files
DAGManSubmission and
Monitoring
(12) DAGMan files
(15) Monitoring
(7) LogicalTransformations
(8) PhysicalTransformations and
Execution EnvironmentInformation
(13) DAG (14) Log FIles
Abstract DAGreduction
(5) Full Abstract DAG (6) Reduced Abstract DAG
MCS Current SateGenerator
MDS
(2) Abstract DAG
Summary
Technology exponentials are changing the shape of scientific investigation & knowledge- More computing, even more data, yet more
networking
The Grid: Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations
Many potential opportunities for application of semantic web technologies to Grid services- OGSA
Partial Acknowledgements
Open Grid Services Architecture design- Karl Czajkowski @ USC/ISI
- Ian Foster, Steve Tuecke @ANL
- Jeff Nick, Steve Graham, Jeff Frey @ IBM Semantic/Cognitive Grid
- Yolanda Gil, Ewa Deelman, Jim Blythe, Tom Russ, Hans Chalupsky
- Conversations with Jim Hendler, Carol Gobel, David DeRoure
Strong links with many EU, UK, US Grid projects Support from DOE, NASA, NSF, Microsoft
For More Information
Grid Book- www.mkp.com/grids
The Globus Project™- www.globus.org
OGSA- www.globus.org/ogsa
Global Grid Forum- www.gridforum.org