grid computing
TRANSCRIPT
Grid Computing
Ian Foster
Mathematics and Computer Science Division
Argonne National Laboratory
and
Department of Computer Science
The University of Chicago
http://www.mcs.anl.gov/~foster
2
[email protected] ARGONNE CHICAGO
Partial Acknowledgements Open Grid Services Architecture design
Carl Kesselman, Karl Czajkowski @ USC/ISI Steve Tuecke @ANL Jeff Nick, Steve Graham, Jeff Frey @ IBM
Grid services collaborators at ANL Kate Keahey, Gregor von Laszewski Thomas Sandholm, Jarek Gawor, John Bresnahan
Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org)
Strong links with many EU, UK, US Grid projects Support from DOE, NASA, NSF, IBM, Microsoft
3
[email protected] ARGONNE CHICAGO
Goals Communicate the purpose, significance,
state, adoption, & future of Grid technology Persuade you that Grid technology
represents an opportunity Grids aren’t (particularly) about science or
servers—themes of virtualization, service discovery, service management, and QoS delivery are universal
Rapid uptake in industry & science represents an exceptional opportunity for impact
4
[email protected] ARGONNE CHICAGO
Overview
Origins: Resource sharing within scientific collaborations Science drivers & science Grid projects Globus Toolkit
Evolution: Resource virtualization Commercial drivers OGSA: Grid meets Web services
5
[email protected] ARGONNE CHICAGO
Overview
Origins: Resource sharing within scientific collaborations Science drivers & science Grid projects Globus Toolkit
Evolution: Resource virtualization Commercial drivers OGSA: Grid meets Web services
6
[email protected] ARGONNE CHICAGO
E-Science: The Original Grid Driver
Pre-electronic science Theorize &/or experiment, in small teams
Post-electronic science Construct and mine very large databases Develop computer simulations & analyses Access specialized devices remotely Exchange information within distributed
multidisciplinary teams Need to manage dynamic, distributed
infrastructures, services, and applications
7
[email protected] ARGONNE CHICAGO www.griphyn.org/chimera
Size distribution ofgalaxy clusters?
1
10
100
1000
10000
100000
1 10 100
Num
ber
of C
lust
ers
Number of Galaxies
Galaxy clustersize distribution
Chimera Virtual Data System+ GriPhyN Virtual Data Toolkit
+ iVDGL Data Grid (many CPUs)
eScience Application: Sloan Digital Sky Survey Analysis
8
[email protected] ARGONNE CHICAGO
•Lift Capabilities•Drag Capabilities•Responsiveness
•Deflection capabilities•Responsiveness
•Thrust performance•Reverse Thrust performance•Responsiveness•Fuel Consumption
•Braking performance•Steering capabilities•Traction•Dampening capabilities
Crew Capabilities- accuracy- perception- stamina- re-action times- SOPs
Engine Models
Airframe Models
Wing Models
Landing Gear Models
Stabilizer Models
Human Models
NASA’s Information Power Grid: Aviation Safety
9
[email protected] ARGONNE CHICAGO
NETWORK
IMAGINGINSTRUMENTS
COMPUTATIONALRESOURCES
LARGE DATABASES
DATA ACQUISITIONPROCESSING,
ANALYSISADVANCED
VISUALIZATION
Life Sciences: Telemicroscopy
10
[email protected] ARGONNE CHICAGO
And Thus: The Grid
“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”
11
[email protected] ARGONNE CHICAGO
Underlying Technical Requirements
Dynamic formation and management of virtual organizations
Online negotiation of access to services: who, what, why, when, how
Configuration of applications and systems able to deliver multiple qualities of service
Autonomic management of distributed infrastructures, services, and applications
12
[email protected] ARGONNE CHICAGO
The Grid World: Current Status Dozens of major Grid projects in scientific &
technical computing/research & education Open source Globus Toolkit™ a de facto standard
for major protocols & services Simple protocols & APIs for authentication, discovery,
access, etc.: infrastructure Information-centric design Large user and developer base Multiple commercial support providers Enabler of numerous tools and applications
Global Grid Forum: community & standards
13
[email protected] ARGONNE CHICAGO
Overview
Origins: Resource sharing within scientific collaborations Science drivers & science Grid projects Globus Toolkit
Evolution: Resource virtualization Commercial drivers OGSA: Grid meets Web services
14
[email protected] ARGONNE CHICAGO
Resource Sharing within “VOs” is Not Unique to Science!
Fragmentation of enterprise infrastructure Driven by cheap servers, fast nets, ubiquitous
Internet, eBusiness workloads Need to configure distributed collections of services
to deliver specified QoS Virtualization
Emerging service infrastructure, utility computing models, economies of scale
Services dynamically instantiated across device spectrum
B2B, B2C, C2C interactions
15
[email protected] ARGONNE CHICAGO
Virtualization andDistributed Service Management
Less capable, integratedLess connected
User service locus
Larger, more integratedMore connected
Dynamically provisioned Device Continuum
Resource &service
aggregation
Delivery of virtualized services with QoS
guaranteesDynamic, secureservice discovery
& composition
Distributed servicemanagement
16
[email protected] ARGONNE CHICAGO
Realizing the PromiseRequires Significant Innovation
Automation of infrastructure operation to achieve economies of scale
Management and component models for service discovery, composition, provisioning
New applications and tools powered by distributed services and resources
Business and service models to support specialization of function
17
[email protected] ARGONNE CHICAGO
Grid Evolution:Open Grid Services Architecture
Refactor Globus protocol suite to enable common base and expose key capabilities
Service orientation to virtualize resources and unify resources/services/information
Embrace key Web services technologies: WSDL as IDL, leverage commercial efforts
Result: standard interfaces & behaviors for distributed system management
18
[email protected] ARGONNE CHICAGO
OGSA System Structure
A standard substrate: the Grid service Standard interfaces and behaviors that address
key distributed system issues The “Grid Service Specification”
… supports standard service specifications Resource management, databases, workflow,
security, diagnostics, etc., etc. Target of current & planned GGF efforts
… and arbitrary application-specific services based on these & other definitions
19
[email protected] ARGONNE CHICAGO
Transient Service Instances “Web services” address discovery & invocation of
persistent services Interface to persistent state of entire enterprise
In Grids, must also support transient service instances, created/destroyed dynamically Interfaces to the states of distributed activities E.g. workflow, video conferencing, distributed data
analysis, workload management Significant implications for how services are
named, discovered, managed, and used
20
[email protected] ARGONNE CHICAGO
OGSI, OGSA, and Web Services
OGSI (I = Infrastructure) Small extensions to WSDL
Nested serviceType & serviceDataDescription
Conventions for naming service instances Handles and references
portTypes for common behavior Instance creation, lifetime management, introspection and
monitoring, registration, notification, …
OGSA (A = Architecture) built on OGSI A collection of Grid service interfaces
Resource description & provisioning Higher-level services: messaging services, logging, etc.
21
[email protected] ARGONNE CHICAGO
The Grid Service =Interfaces/Behaviors + Service Data
Servicedata
element
Servicedata
element
Servicedata
element
Implementation
GridService(required)Service data access
Explicit destructionSoft-state lifetime
… other interfaces …(optional) Standard:
- Notification- Authorization- Service creation- Service registry- Manageability- Concurrency
+ application-specific interfaces
Binding properties:- Reliable invocation- Authentication
Hosting environment/runtime(“C”, J2EE, .NET, …)
22
[email protected] ARGONNE CHICAGO
Service Data A Grid service instance maintains a set of service data
elements Described in WSDL extension XML element encapsulated in standard container: name,
type, lifetime, etc. Includes basic introspection information, interface-specific
data, and application state Pull and push models for information query
GridService::FindServiceData operation Pull: queries this information via extensible query language
NotificationSource::SubscribeServiceData Push: Subscribe to notification of changes to information
23
[email protected] ARGONNE CHICAGO
Notification Interfaces NotificationSource for client subscription
Subscription expression describes which service data element changes are of interest
Creates a subscription manager service Manages the lifetime and properties of subscription
NotificationSink for asynchronous delivery of notification messages
Simple, flexible base with wide variety of uses Dynamic discovery/registry services, monitoring, application
error notification, etc. Intermediaries: filter, aggregate, archive, et.c Can integrate commercial messaging services
24
[email protected] ARGONNE CHICAGO
Grid Service Example:Database Service
A DBaccess Grid service will support at least two portTypes GridService DBaccess
Each has service data GridService: basic introspection information, lifetime,
… DBaccess: database type, query languages
supported, current load, …, … Maybe other portTypes as well
E.g., NotificationSource
GridService DBaccess
DB info
Name, lifetime, etc.
25
[email protected] ARGONNE CHICAGO
Lifetime Management GS instances created by factory or manually;
destroyed explicitly or via soft state Negotiation of initial lifetime with a factory
(=service supporting Factory interface) GridService interface supports
Destroy operation for explicit destruction SetTerminationTime operation for keepalive
Soft state lifetime management avoids Explicit client teardown of complex state Resource “leaks” in hosting environments
26
[email protected] ARGONNE CHICAGO
Factory
Factory interface’s CreateService operation creates a new Grid service instance Reliable creation (once-and-only-once)
CreateService operation can be extended to accept service-specific creation parameters
Returns a Grid Service Handle (GSH) A globally unique URL, resolves to GSR Uniquely identifies the instance for all time Based on name of a handle resolver
Or Grid Service Reference (GSR)
28
[email protected] ARGONNE CHICAGO
Example:Transient Database Services
GridService
DBaccessFactory
Factory info
Instance name, etc.
GridService Registry
Registry info
Instance name, etc.
GridService DBaccess
DB info
Name, lifetime, etc.
GridService DBaccess
DB info
Name, lifetime, etc.
“What services can you create?”
“What database services exist?”
“Create a database service”
29
[email protected] ARGONNE CHICAGO
OGSA Design & Implementation
OGSI (I=Infrastructure) WG in GGF defining core Grid service specification (At least) three implementation efforts
Globus Toolkit => GT3 (alpha end 2002) GT3 Core: Grid service specification GT3 Base: Globus Toolkit behaviors CIM resource model, GRAM-2 SLA negotiation,
database services, … Other GGF WGs address OGSA security, OGSA-
compliant database services, etc.
30
[email protected] ARGONNE CHICAGO
Recap: Goals
Communicate the purpose, significance, state, adoption, & future of Grid technology
Persuade you that Grid technology represents a significant opportunity Grids aren’t only (or particularly) about
science and servers—themes of virtualization, service discovery, service management, and QoS delivery are universal
Rapid uptake in industry & science represents an exceptional opportunity for impact
31
[email protected] ARGONNE CHICAGO
For More Information
The Globus Project™ www.globus.org
Context & research articles www.mcs.anl.gov/~foster
Open Grid Services Architecture www.globus.org/ogsa
Global Grid Forum www.gridforum.org Edinburgh, July 22-24 Chicago, Oct 15-17
32
[email protected] ARGONNE CHICAGO
OGSA Implementation
Hosting EnvironmentResource virtualization
and QoS support
StandardOGSI
container
Webservicesvarious
OGSI/OGSA Interfacesservice description,
service provisioning, …
2) to enable virtualization via Service description Service provisioning
OGSI/OGSA Interfacesservice description,
service provisioning, …
3) Standard container avoids implementing OGSI features in every service instance
StandardOGSI
container
Hosting EnvironmentResource virtualization
and QoS support
Webservicesvarious
1) OGSA builds on infrastructure Plumbing: WSDL, WS-Security, WS-
Routing/Referral, reliable messaging, transactions, etc.
Hosting environments
33
[email protected] ARGONNE CHICAGO
Building an OGSI Container
Service data mgmt, query, subscription Container should provide simple interface for
interacting with an instance’s implementation to get and manage dynamic service data
Service instance = CLR object
Container should handle query processing .NET support for XPath & Xquery allows for rich functionality
Container manages notification subscriptions, and drives asynchronous notification messages
Soft-state lifetime management Soft-state registration