facultatea de automatica si calculatoare universitatea “politehnica“ din bucuresti current...
Post on 19-Dec-2015
220 views
TRANSCRIPT
Facultatea de Automatica si Calculatoare
Universitatea “Politehnica“ din Bucuresti
Current trends in Grid computing
Dobre Ciprian Mihai
cipsm {at} cs.pub.ro
““The Internet is about getting computers to talk together; The Internet is about getting computers to talk together; grid computing is about getting computers to work together.” grid computing is about getting computers to work together.”
(from IBM’s Grid definition)(from IBM’s Grid definition)
Outline of the presentation
What is Grid computing – sorting out the alphabet soup. Impact of Grid computing to science. CERN as a driving force in Grid computing. Grids – where to ?
What is Grid ?
Many definitions of Grid computing Term coined as analogy to electrical power grid According to Ian Foster, the “father of grid computing”, the
term grid has been hijacked to “embrace everything from advanced networking to artificial intelligence”
Marketers are applying grid labels to all sorts of products and services, adding to the confusion and hype
“From the wide ranging definitions of Grid, to the volume of standards bodies and organizations -- it can be a real challenge to distinguish the significant developments from the hype.” (Ian Foster, 2005)
Electrical Power Grid Computing Grid
Pervasive Everywhere, wall socket Everywhere, any “net-thing”
Transparent Power “just happens” Power “just happens”
Infrastructure Power stations, transformers, powerlines, transmission hubs CPUs, servers, storage, networks, archives, middleware
Utility Pay-for-use, accounting, reporting, settlement Pay-for-use, accounting, reporting, metering
Ian Foster’s Evolving definitions
“A computational grid is a hardware and software infrastructure that provides dependable, pervasive, and inexpensive access to high-end computing capabilities”
Ian Foster and Carl Kesselman, editors, “The GRID: Blueprint for a New Computing Infrastructure”, Morgan-Kaufman Publishers, 1999.
“The grid infrastructure consists of protocols, application programming interfaces, and software development kits to provide authentication, authorization, and resource location/access”
2001: Foster, Kesselman, Tuecke: “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, http://www.globus.org/research/papers.html
“The grid integrates services across distributed, heterogeneous, dynamic ‘virtual organizations’ formed from the disparate resources within a single enterprise and/or from external resource sharing and service provider relationships in both e-business and e-science”
2002: Foster, Kesselman, Nick, Tuecke: “The Physiology of the Grid”, http://www.globus.org/research/papers/ogsa.pdf
GGF: “A system that is concerned with the integration, virtualization, and management of services and resources in a distributed,
heterogeneous environment that supports collections of users and resources (virtual organizations) across traditional
administrative and organizational domains (real organizations).”
GGF: “A system that is concerned with the integration, virtualization, and management of services and resources in a distributed,
heterogeneous environment that supports collections of users and resources (virtual organizations) across traditional
administrative and organizational domains (real organizations).”
CoreGRID: “A fully distributed, dynamically reconfigurable, scalable and autonomous infrastructure
to provide location independent, pervasive, reliable, secure and efficientaccess to a coordinated set of services encapsulating and virtualizing resources
(computing power, storage, instruments, data, etc.) in order to generateknowledge.”
CoreGRID: “A fully distributed, dynamically reconfigurable, scalable and autonomous infrastructure
to provide location independent, pervasive, reliable, secure and efficientaccess to a coordinated set of services encapsulating and virtualizing resources
(computing power, storage, instruments, data, etc.) in order to generateknowledge.”
IDC: “Set of independent computers combined intounified system through systems software and
networking technologies”
IDC: “Set of independent computers combined intounified system through systems software and
networking technologies”
Gartner: “a collection of resources owned by multiple organizations coordinated in such a way as to allow them
to solve a single common problem.”
Gartner: “a collection of resources owned by multiple organizations coordinated in such a way as to allow them
to solve a single common problem.”
IBM: “ability, using a set of open standardsand protocols, to gain access to applications and data,
processing power, storage capacity, and a vast array of othercomputing resources over the Internet”
Grid computing is a network of computation: tools andprotocols for coordinated resource sharing and problem
solving among pooled assetsApplication processing, distributed across multiple
locations, and interconnected through a shared networksuch as the Internet
IBM: “ability, using a set of open standardsand protocols, to gain access to applications and data,
processing power, storage capacity, and a vast array of othercomputing resources over the Internet”
Grid computing is a network of computation: tools andprotocols for coordinated resource sharing and problem
solving among pooled assetsApplication processing, distributed across multiple
locations, and interconnected through a shared networksuch as the Internet
SUN: “A way of managing and dynamically sharingdisparate sets of resources”
A hardware and software infrastructure that connectsdistributed computers, storage devices, databases, and
software applications through a network, and is managedby distributed resource management software
A dependable, universal information infrastructure thatbuilds on the power of the Internet and enables more
efficient computation, collaboration, and communication
SUN: “A way of managing and dynamically sharingdisparate sets of resources”
A hardware and software infrastructure that connectsdistributed computers, storage devices, databases, and
software applications through a network, and is managedby distributed resource management software
A dependable, universal information infrastructure thatbuilds on the power of the Internet and enables more
efficient computation, collaboration, and communication
“ A Grid is a large, heterogeneous, system that allows sharing and coordinating resources
in a dependable and pervasive manner “
“ A Grid is a large, heterogeneous, system that coordinates resources spread over wide ares ““ A Grid is a heterogeneous system that allowsmultiple entities to share and use resources,
under various administrative policies,offering a transparent access to the user “
“ A Grid is a heterogeneous system spreadover a wide geographical area, which allowsmultiple entities to share and use resources,
under various administrative policies,offering a transparent access to the user, through
the use of consistent access protocols and interfaces “
Why so many definitions?
Computer science and software engineering sometimes do not have definitions as strict as those in the fields of physics or mathematics – this “lack of definitions” leads to many Grid researchers or people working with Grid technology having different views on what a Grid is.
Hardware discrepancies: for some a local cluster with a middleware system on top is a Grid whereas others believe that a wide-are network connection has to be involved.
Software problems: What actually makes a piece of software a “Grid software”? Is any kind of middleware using Grid security already Grid software?
Due to the recent advanced in Web and Grid service technologies, where to draw the line between Web services and Grid services?
So what is Grid after all?
In this Soup of grid definitions there are two that were widely accepted by the community:
I. Foster, Research view: “A Grid is a system that (1) coordinates resources that are
not subject to centralized control (2) using standard, open, general-purpose protocols and interfaces (3) to deliver nontrivial qualities of service”
A. Grimshaw, Industry view: “From a hardware perspective, a Grid is a collection of
distributed resources connected by a network. From a user perspective a Grid gathers together resources and makes them accessible in a secure manner to users and applications”
Describing the elephant
A Grid infrastructure must provide a set of technical capabilities:1. Resource modeling – describes available resources, their capabilities, and
the relationships between them to facilitate discovery, provisioning, and quality of service management.
2. Monitoring and notification – provides visibility into the state of resources to enable discovery and maintain quality of service. Logging of significant events and state transitions is also needed to support accounting and auditing functions.
3. Allocation – Assures quality of service across an entire set of resources for the lifetime of their use by an application. This is enabled by negotiating the required level(s) of service and ensuring the availability of appropriate resources through some form of reservation—essentially, the dynamic creation of a service-level agreement.
4. Provisioning, life-cycle management, and decommissioning - enables an allocated resource to be configured automatically for application use, manages the resource for the duration of the task at hand, and restores the resource to its original state for future use.
5. Accounting and auditing - tracks the usage of shared resources and provides mechanisms for transferring cost among user communities and for charging for resource use by applications and users.”
6. In addition to that security is an important aspect.
Foster, Tuecke, “Describing the elephant: the different faces of IT as services”, ACM Queue, 2005.
The two key Grid computing groups
The Globus Alliance (www.globus.org) Composed of people from: Argonne National Labs, University of
Chicago, University of Southern California Information Sciences Institute, University of Edinburgh and others.
OGSA/I standards initially proposed by the Globus Group Based off papers “Anatomy of the Grid” & “Physiology of the
Grid”
The Global Grid Forum (www.ggf.org) First meeting in June of 1999, Based off the IETF charter Heavy involvement of Academic Groups and Industry (e.g. IBM
Grid Computing, HP, United Devices, Oracle, UK e-Science Programme, US DOE, US NSF, Indiana University, and many others)
Meets three times annually Solicits involvement from industry, research groups, and
academics
More on Grids
The Grid relies on advanced software, called middleware, which ensures seamless communication between different computers and different parts of the world.
The Grid search engine finds the data the scientist needs, but also the data processing techniques and the computing power to carry them out.
It then distribute the computing task to wherever in the world there is spare capacity, and send the result to the scientist.
Why use the Grids? Industrial and academic partners form an “extended enterprise”
in which resources are intrinsically distributed, and only partially shared.
Partners may be prepared to share data, but not the hardware and proprietary software that produces the data.
Why Grid computing now? Let us look at the evolution of ICT
Grid-like Vision
In 1969, Leonard Kleinrock, one of the chief scientists of the original ARPA project which seeded the Internet, wrote: "As of now, computer networks are still in their infancy,
but as they grow up and become sophisticated, we will probably see the spread of "computer utilities", which, like present electric and telephone utilities, will service individual homes and offices across the country“
Despite major advances in hardware and software systems over the past 35 years, we are yet to realize this vision. How far are we still from delivering computing as a utility? Let us look into the ICT evolution and project the future.
Computing and Communication Technologies Evolution: 1960-
2010!
* Sputnik
1960 1970 1975 1980 1985 1990 1995 2000
* ARPANET
* Email* Ethernet
* TCP/IP* IETF
* Internet Era * WWW Era
* Mosaic
* XML
* PC Clusters* Crays * MPPs
* Mainframes
* HTML
* W3C
* P2P
* Grids
* XEROX PARC wormCO
MP
UTIN
GC
om
mu
nic
ati
on
* Web Services
* Minicomputers
* PCs
* WS Clusters
* PDAs* Workstations
* HTC
2010
* e-Science
* Computing as Utility
* e-Business
* SocialNet
ControlCentralised Decentralised
Computing is Scaling: Towards Inter-Planetary Level
2100
2100 2100 2100 2100
2100 2100 2100 2100
Personal Device SMPs or SuperComputers
LocalCluster
GlobalGrid
SERV ICES
+
PERFORMANCE
Inter PlanetGrid
•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe
Administrative Barriers
EnterpriseCluster/Grid
A little bit more…
Benefits for Science: More effective and seamless collaboration of dispersed
communities, both scientific and commercial Ability to run large-scale applications comprising thousands of
computers, for wide range of applications. Transparent access to distributed resources from your desktop,
or even your mobile phone The term “e-Science” has been coined to express these
benefits – the application domain “Science” of Grid & Web Impact : e-Science
From the EPSRC e-Science web site:
"In the future, e-Science will refer to the large-scale science that will increasingly be carried out through distributed global collaborations enabled by the Internet. Typically, a feature of such collaborative scientific enterprises is that they will require access to very large data collections, very large scale computing resources and high performance visualisation back to the individual user scientists."
Healthy, Wealthy, and Wise?
e-Health: electronic patient records, distributed and/or remote diagnosis, collaborative surgical planning.
e-Business: streamline, distribute, and enhance business processes.
e-Commerce: use the Grid as a marketplace for both traditional and innovative goods and services.
e-Learning: remove barriers to education and training.
Grid applications for Science: Medical/Healthcare (imaging, diagnosis and treatment ). Bioinformatics (study of the human genome and proteome to
understand genetic diseases). Nanotechnology (design of new materials from the molecular
scale). Engineering (design optimization, simulation, failure analysis and
remote Instrument access and control). Natural Resources and the Environment (weather forecasting,
earth observation, modeling and prediction of complex systems)
Grid and Web Services Convergence
Definition of Web Service Resource Framework (WSRF) makes explicit distinction between “service” and stateful entities acting upon service i.e. the resources
Means that Grid and Web communities can move forward on a common base!!!
Grid
OGSi
GT2
GT1
Web HTTPWSDL,
SOAP
WS-*
Have beenconverging
WSRF
Started far apart in
applications &
technology
XML
BPEL
WS-I Compliant
TechnologyStack
Grid and Web Services
The Globus Grid Forum (GGF) standard was (2004) divided into: Open Grid Services Architecture (OGSA)
Defines standard mechanisms for creating, naming, and discovering Grid service instances.
Addresses architectural issues relating to interoperable Grid services. An open, service-oriented architecture (SOA): resources as first-class
entities, dynamic service/resource creation and destruction Built on a Web service infrastructure Resource virtualization at the core Build grids from small number of standards-based components
(replaceable, coarse-grained) Customizable: Support for dynamic, domain-specific content… within the
same standardized framework Described in “The Physiology of the Grid”
http://www.globus.org/research/papers/ogsa.pdf Open Grid Services Infrastructure (OGSI)
It was based upon Grid Service specification. It specifies the way clients interact with a grid service (service invocation management, data interface, security interface, ...).
In the new draft (2005-06) some mandatory specifications of OGSI are merged with OGSA and new WSRF is introduced (GT4)
WSRF : Web Services Resource Framework : defines a generic and open framework for modeling and accessing stateful resources using web services
The core elements of the Open Grid Services Architecture
This layer eliminated in recent version of standard
Pre-GT4
GT4
Virtualizing Resources
Resources
Webservices
AccessAccess
StorageStorage SensorsSensors ApplicationsApplications InformationInformationComputersComputers
Resource-specific InterfacesResource-specific Interfaces
Common Interfaces
Type-specific interfaces
A Service-Oriented Grid
Virtualized resources
Grid middleware
services
Brokering Service
Brokering Service
Registry Service
Registry Service
DataService
DataService
CPU ResourceCPU ResourcePrinter Service
Printer Service
Job-Submit Service
Job-Submit Service
ComputeService
ComputeService
No
tify
Ad
vertise
ApplicationService
ApplicationService
Global Grid Community
CERN is the world's largest particle physics centre
Particle physics is about: elementary particles which all matter in the Universe is made of fundamental forces which hold matter together
Particles physics requires: special tools to create and study new particles
With its 27 km circumference, the LHC accelerator will be the largest superconducting installation in the world.
CERN?
CERN is:-~ 2500 staff scientists (physicists, engineers, …)- Some 6500 visiting scientists (half of the world's particle physicists)
They come from 500 universities representing
80 nationalities.
Computing @ CERN
Latest trend is to federate national Grids to achieve a global Grid infrastructure – High Energy Physics is a driving force to this.
High-throughput computing based on reliable “commodity” technology
LHC Data Analysis requires a computing power equivalent to ~100,000 of today’s fastest PC processors !
More than 2500 dual processor PCs About 3 million Gigabytes of data on disk and tapes PROBLEM: nowhere near enough!• SOLUTION: use the Grid to unite computing resources of particle
physics institutes around the world.• CERN leads two major global Grid projects:
• WLCG: World-wide LHC Computing Grid Collaboration• EGEE: Enabling Grid for E-sciencE project for all sciences
WLCG: All the Institutions participating in the provision of the Worldwide LHC Computing Grid with a Tier-1 and/or Tier-2 Computing Centre form the WLCG Collaboration.
The LHC Computing Grid project launched a service with 12 sites in 2003. Today 200 sites in 30 countries with 16,000 PCs.
Computing @ CERN
The LCG architecture consists of an agreed set of services and applications running on the Grid infrastructures provided by the LCG partners.
These infrastructures at the present consist of those provided by the Enabling Grids for E-sciencE (EGEE) project in Europe, the Open Science Grid (OSG) project in the U.S.A. and the Nordic Data Grid Facility in the Nordic countries.
Grid3 was the start-up of OSG The LCG Project builds and maintains computing infrastructure for
LHC experiments Original (’02) LCG plan: “The LCG is not a middleware project” Was to be delivered... too little, too late Feature set, performance, scalability disappointing New (’04) plan: Middleware “re-engineering” as part of the LCG
program, in collaboration with EGEE
EGEE-II: Fast description of the project
EGEE launched in 2004, already supports 20 applications in six scientific domains (biomedicine, geophysics, quantum chemistry…)
EGEE brings together scientists and engineers of 90 institutions
In over 30 countries worldwide To provide seamless GRID
infrastructure for e-Science Available 24 h/day x 7days/week Funded by EU (European Commission) Two original scientifically fields: HEP
and Life Sciences; but it integrates many other fields: from Geology up to Computing Chemistry
Infrastructure: 30.000 CPUS , 5 PBbytes storage, 200 sites in 39 countries, 60 Virtual Organizations
Maintains 10.000 concurrent jobs on average
Computing @ CERN
Three Generations of Grid
Source: Charlie Catlett
StandardizationStandardization is key for third-generation grids! is key for third-generation grids!StandardizationStandardization is key for third-generation grids! is key for third-generation grids!
• Local “metacomputers“– Distributed file systems– Site-wide single sign-on
• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept
• Local “metacomputers“– Distributed file systems– Site-wide single sign-on
• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept
• Utilize software services and communications protocols developed by grid projects:
– Condor, Globus, UNICORE, Legion, etc.
• Need significant customization to deliver complete solution• Interoperability is still very difficult!
• Utilize software services and communications protocols developed by grid projects:
– Condor, Globus, UNICORE, Legion, etc.
• Need significant customization to deliver complete solution• Interoperability is still very difficult!
• Common interface specifications support interoperability of discrete, independently developed services
• Competition and interoperability among applications, toolkits, and implementations of key services
• Common interface specifications support interoperability of discrete, independently developed services
• Competition and interoperability among applications, toolkits, and implementations of key services
We are here!
We are here!
Grids – Where to ?
The commercial interest in Grids systems and related technologies is increasing.
Companies such as Sun Microsystems, IBM, Oracle, Intel, Microsoft, HP show particular interest in getting a piece of the $12 billion market predicted by IPC for 2007 (according to IDC).
0
2000
4000
6000
8000
10000
12000
14000
2003 2004 2005 2006 2007
Year
Mill
ion
Grids – Where to ?
After the year 2007, business popularity of Grid computing is expected to accelerate: Especially, the financial services and ERP services is
expected to take major parts in the expense (Source: Insight Research Corp.)
Bill
ion
s
Grids – Where to ?
An interesting prediction (the 451 Group analysts) is that grid technology will be slowly absorbed into enterprise fabrics…
One consequence for grid computing might be that term grid computing "will become both more relevant and less used […] It will be more relevant as grids are used to support far more than HPC tasks, but less used as vendors seek to be associated with far more activity, and far higher up the stack, than grid computing."
IBM and Oracle could drop "grid" from their products in favour of a broader term, while Microsoft has made it very clear that it will not use the term “grid”.
In the new era of Grid computing grids must support automated data, storage and service activities just as capably as handling computational tasks.
These challenges are being addressed by a new paradigm called “Grid 2.0”
Grids – Where to ?
Grid 1.0 – concerned with the virtualization, aggregation and sharing or compute resources
Grid 2.0 – focused on the virtualization, aggregation and sharing of all compute, storage, network and data resources
The key term is “virtualization” (encapsulation behind a common interface of diverse implementations) is being driven by the need to various enterprises to create a virtual resource market to allocate resources based on business demand.
Virtualization introduces a layer of abstraction: instead of having to snoop out what resources are available and try to adapt a problem to use them, a user can describe a resource environment (virtual workspace) and expect it to be deployed on the grid. The mapping between the physical resources and the virtual workspace will be handled using virtual machines, virtual appliances, distributed storage facilities and network overlays (“virtual grids”).
The promise is that in Grid 2.0 the resources will be easier to define, test, install, transport and adjust on demand.
CSComputer Science
Department
Web 2.0: Evolution Towards a Read/Write Platform
Web 1.0(1993-2003)
Pretty much HTML pages viewed through a browser
Web 2.0(2003- beyond)
Web pages, plus a lot of other “content” shared over the web, with more interactivity; more like an application than a
“page”
“Read” Mode “Write” & Contribute
“Page” Primary Unit of content
“Post / record”
“static” State “dynamic”
Web browser Viewed through… Browsers, RSS Readers, anything
“Client Server” Architecture “Web Services”
Web Coders Content Created by… Everyone
“geeks” Domain of… “mass amatuerization”
CSComputer Science
Department
Web 2.0 By Example
Web 1.0 Napster Britannica On Line Akamai MP3.com Double Click Content Management
Web 2.0 Google Wikipedia BitTorrent iTUNES or Napster Adsense Wikis
Tim O’Reilly
CSComputer Science
Department
Google Earth™ a Mega API for Web 2.0
Illustrates the Benefits of SOA and GRID with a Web 2.0 Delivery Model
• Distributed, re-usable core services on shared infrastructure
• Shared data
• Exposed interfaces
• Application is streamed to client and works offline
CSComputer Science
Department
Google Earth™ a Mega API for Web 2.0
CSComputer Science
Department
Wikipedia is a Collaborative Dictionary Being Edited in
Realtime by Anyone
Grid 2.0 EmergingGrid 2.0 Emerging
Grid 1.0Grid 1.0Compute Intensive Compute Intensive Cycle AggregationCycle Aggregation
SOASOASoftware Services Software Services with SLA & QoS with SLA & QoS
MetricsMetrics
Virtualization Virtualization Consolidation of Consolidation of
ResourcesResources
Grid 2.0*Grid 2.0* Virtualized Compute, Virtualized Compute,
Storage, Network, DataStorage, Network, Data
Service OrientedService Oriented
Policy Driven AutomationPolicy Driven Automation
Distributed across Distributed across firewallsfirewalls
Parallel, stateless, Parallel, stateless, stateful and transactional stateful and transactional appsapps
*The 451 Group: 'grid 2.0' is focused on the virtualization, aggregation and sharing of all *The 451 Group: 'grid 2.0' is focused on the virtualization, aggregation and sharing of all compute, storage, network and data resources. It is both Service-oriented and automated.compute, storage, network and data resources. It is both Service-oriented and automated.
Virtualization
Virtualization covers both, data (flat files, databases etc.) and computing resources. Grid as workflow virtualization — the Grid computing services are used to execute and
manage processes across multiple compute platforms. Data Grid as data virtualization — the management of shared collections
independently of the remote storage systems where the data is stored. Semantic Grid as information virtualization — the ability to reason on inferred
attributes from multiple independent information repositories. Name space virtualization, logical names for resources, users, files, and metadata that
are independent of the name spaces used on the remote resource. Trust virtualization, the ability to manage authentication and authorization
independently of the remote resource. Constraint virtualization, the ability to manage access controls independently of the
remote resource. Access virtualization, the ability to port an arbitrary access mechanism on top of the
Grid middleware. For Data Grids, this is the ability to support access through multiple loadable libraries, Java, Digital libraries, workflow actors, Web browsers, etc.
Network virtualization, the ability to manage transport in the presence of network devices such as firewalls, load levelers, private virtual networks. This typically requires multiple protocols to support client-initiated versus server-initiated I/O, bulk operations versus single-file operations.
Latency management, the ability to minimize the number of messages sent over wide area networks. Examples include execution of procedures at the remote resource when the complexity (ratio of operations to bytes transmitted) is sufficiently small. The standard case is data filtering or sub-setting.
Federation, the ability to interoperate across multiple grid environments. This requires the ability to share logical name spaces, and Shibboleth-style authentication. Grids establish trust mechanisms to allow assertions about the authenticity of an individual to be verified from the “home” Grid.
So, are we there yet ?
Will the Grid be available to all of you ? Hard to predict…
Jules Piccard, a professor at the University of Basel, installed the first telephone in the city, around 1880, between his home and his institute. He showed it proudly to other scientists and got the comment: “Looks very good, but I doubt it will ever have any practical use”.
"The world will only need five computers"attributed to Thomas J. Watson, IBM
"640 kilobytes is all the memory you will ever need"attributed to Bill Gates, Microsoft
"There is absolutely no need for a computer in the home"attributed to Ken Olsen, DEC (once a leading minicomputer manufacturer)
So, are we there yet ?
The complete success of the Grid hype depends on at least three conditions:
The Grid can be considered a success when there are no more “Grid papers”, but only a footnote in the work that states, “This work was achieved using the Grid”.
The Grid can be considered a success when supercomputer centres don't give a user the choice of using their machines or using the Grid, they just use the Grid.
The Grid can be considered a success when a SuperComputing demo can be run any time of the year. We are not yet there…
What’s holding us ?
Organizational politics act very much like a barrier to implementing Grid computing:
“server-hugging” – organizations have a sense of ownership over the resources bought or allocated for their use.
unrealistic expectations from Grid computing – marketing departments have run amuck and have marketed the grid “nirvana” and not the grid that exists and is possible today.
perceived loss of control or access over resources. loss or reduction of budget dollars. lack of data security among departments. fear of external data leaks, reduced priority of projects - sometimes users believe that they
need dedicated IT resources to complete their work accurately and efficiently.
risks associated with enterprise-wide deployment - how do different geographies and cultures come together to agree on global priorities, configurations, standards, and policies.
In the end…
One of the biggest fears for Grid computing is that it might be seen as today’s sexy technology that will quickly get replaced by tomorrow’s sexy technology.
The Grid researchers and technologists have to start to point to results/applications that utilize the Grid to solve problems or enable new applications that would have be unachievable without the Grid.
Contemporary Grid implementations are still far from initially described image and from being widely adopted.
Grid computing in pictures
Thanks to GridCafe (http://gridcafe.web.cern.ch/gridcafe - i strongly recommend that you also visit this link), it is now MOVIE time.
Thank you !
Questions? Observation?Questions? Observation?
Additional slides
Grid characteristics
Collaboration - Grid is sharing of resources in a distributed fashion. A Grid spans multiple administrative domains seamlessly.
Aggregation - A Grid is more than the sum of all parts. A Grid aggregates many resources and therefore provides an aggregation of the capacity of the individual resources into a higher capacity virtual resource. The capability of individual resources is preserved. As a consequence, from a global standpoint the Grid enables running larger applications faster (aggregation capacity), while from a local standpoint the Grid enables running new applications
Virtualization – Grid services are often provided with a certain interface that hides the complexity of the underlying resources. Virtualization provides an abstract “layer” between clients and resources, Therefore, a Grid provides the ability to virtualize the sum of parts into a singular wide-area programming model.
Grid characteristics
Service orientation - Grids provide services, following the concept of a service orient architecture. In the widest sense all large scale collections of services can be viewed as Grids.
Heterogeneity - A Grid typically consists of heterogeneous computing resources, i.e. there is a variety of different hardware and software components with different performance and latency characteristics.
Decentralized control - components are under control of multiple entities, i.e. the key difficulties in Grids lay exactly in not having a single ‘owner’ of the whole system. One of the requirements of a Grid is the use of distributed control mechanisms
Standardization and interoperability - A Grid promotes standard interface definitions for services that need to interoperate to create a general distributed infrastructure to fulfill users’ tasks and provide user level utilities. Grid is exposing the need for increased levels of integration of distinct technologies and for increased agreements in the standardization of services. The success of the implementation of the Grid very much depends on these aspects. Furthermore, the Grid should provide uniform access to heterogeneous resources through virtualization.
Grid characteristics
Access transparency - The Grid should allow its users to access the computing infrastructure without having to be intimately aware of the underlying architecture or network topology]. This is sometimes considered the most distinctive aspect of Grid Computing, that is, the levels of transparency provided for the end-user, through the virtualization of resources.
Scalability - Even if Grid implementations and infrastructures sometimes do not solve a new problem, it is often the scale of data, resources and users that contributes to the additional complexity of a Grid.
Reconfigurability - A Grid should be “dynamically reconfigurable” (CoreGRID definition).
Security - Grid security is one of the first things that real Grid users have to deal with and therefore is essential for any Grid software system that spans multiple administrative domains.
Application support – Applications should also be part of the Grid and the whole Grid environment (where for environment I mean the hardware, middleware, and applications) should be data-driven. In particular, it should be able to react to changes of the system and application behaviors captured by application and system data.
Grid characteristics
Computing model - a Grid supports several computational models (e.g., batch, interactive, distributed and parallel computing. . . ).
Licensing model - Since Grids originate from the academic community, there is a global emphasis on open source software, which is also followed by several companies that are involved in Grid development.
Procedures and policies - Grid users and service providers interact with each other in a similar way like on the open market where certain rules have to be followed. Therefore, procedure and polices need to be in place to allow for (coordinated) sharing of resources.
Auditing - Tracking the usage of shared resources and providing mechanisms for transferring cost among user communities and for charging for resource use by applications and users.
Comparison of Middleware Technologies
MiddlewareProperty UNICORE Globus Legion Gridbus
Focus High level Programming models
Low level services High level Programming models Abstractions and market models
Category Mainly uniform job submission and monitoring
Generic computational Generic computational Generic computational
Architecture Vertical multi tiered system Layered and modular toolkit Vertically integrated system Layered component and utility model
Implementation Model
Abstract Job Object Hourglass model at system level Object-oriented metasystem Hourglass model at user level
Implementation Technologies
Java C and Java C++ C, Java, C# and Perl
Runtime Platform Unix Unix Unix Unix and Windows with .NET
Programming Environment
Workflow environment Replacement libraries for Unix & C libraries. Special MPI library (MPICH –G), CoG (Commodity Grid) kits in Java, Python, CORBA, Matlab, Java Server Pages, Perl and Web Services
Legion Application Programming Interfaces (API).Command line utilities
Broker Java APIXML-based parameter-sweep language Grid Thread model via Alchemi.
Some Users and Applications
EuroGrid], Grid Interoperability Project, OpenMolGrid and Japanese NAREGI.
AppLeS, Ninf], Nimrod-G, NASA IPG, Condor-G, Gridbus Broker, UK eScience Project], GriPhyN], and EU Data Grid.
NPACI Testbed, Nimrod-L, and NCBioGrid. Additionally, it has been used in the study of axially symmetric steady flow and protein folding applications.
ePhysics, Belle Analysis Data Grid], NeuroGrid], Natural Language Engineering, HydroGrid, and Amsterdam Private Grid].
Globus Toolkit Components
Globus Common Runtime
Python Web Services Core Allows one to create WSRF-
compliant web services in Python
C Web Services Core Allows one to create WSRF-
compliant web services in C Java Web Services Core
WSRF APIs in Java C common libraries
C Abstraction layer for Globus data types, libc, etc
XIO : Extensible IO Superset of basic file I/O
library open/close/read/write/etc
Supports multiple wire protocols transparently
TCP/UDP/File/HTTP/GSI/GSSAPI_FTP/telnet/queuing
Globus XIO
Open file for reading/writing inFile = fopen( file, “w+” )
Do some reading… fread(buffer,1,sizeof(buffer
),inFile) Do some writing…
fprintf(inFile, “%s\n”, “HELLO!” )
Close the file fclose( inFile )
Disk Storage
[Mountain View, CA]
Student Workstation [Denton, TX]
Globus Information Services
WebMDS Allows one to view monitoring
information about grid resources from a web browser
Index Collects monitoring and discovery
information grid resources. Publishes the information to a single
point so other resources/peoples can discover resources
Trigger Collects various pieces of data from
grid resources Can be configured to perform actions Ex : when a disk is 80% full, send an
email to the administration staff Monitoring and Discovery [MDS2]
Provides method to publish and discover resources on the grid.
Also allows the collection of resource status and configuration information
Deprecated component
Globus Indexing Service
Globus Index Resource Query
Computational
Resources
Database
Resources
Archival Resources
Storage Resources
Globus Execution Management
Grid Resource Allocation and Management [GRAM]
WSRF compliant device to submit, monitor, and cancel jobs on grid computing resources
Not a scheduler, but rather communicates with other, local schedulers
Pre-WS Grid Resource Allocation and Management
Same as above… not a web service Community Scheduler Framework
WSRF compliant meta-scheduler. Actually schedules jobs to other
batch scheduler. Supports advanced reservations and
advanced scheduling policies Grid Telecontrol Protocol
WSRF protocol for telecontrol Ex : focusing an electron microscope
remotely Workspace Management
Dynamically create and manage workspaces
Globus GRAM
GRAM User Job Submission
PBS
SGE
LSF Condor
Site A
Site B
Site C
Site D
Geographically Disparate Computational Resources
Globus Data Management
Data Replication Allows for the local replication of
pertinent data across grid environments Commonly used files are replicated
locally to reduce transfer delays OGSA-DAI
Supports the exposure of data resources such as relational databases or XML databases onto the grid
Single point of query for multiple databases
Reliable File Transfer Handles third-party messages to control
GridFTP transfers Submission of transfer requests
GridFTP High performance, reliable data transfer
protocol for high bandwidth, wide area networks.
Used to perform the data transfer test that became the LAN Speed Record
Replica Location Allows discovery and registration of
data replicas on the grid Maintains the correlation between
logical names and target names
Traditional FTP
Single Client/Server connection
Single data stream Limited by computational
resources and network bandwidth
Very inefficient
Gigabit E
thernet
Data Channel
Globus GridFTP
Single Server/Client
Gigabit E
thernet
Data Channel
Multiple Server/Client [single file]
Storage
StorageEach machine
transfers ¼ of the file
Globus Security
Community Authorization Allows a virtual organization to express
policy regarding resources across sites Despite the local authorization, granting
and revoking access to resources is possible
Delegation Allows the sharing of a single credential
across multiple invocations of services I need to submit multiple jobs, now I can
use the same certificate for each one Authentication/Authorization
Message & Transport Level Security SSL/TLS/X.509 encryption standard for
message traffic Authorization Framework
Provides multiple different authorization mechanisms : gridmap, SAML, NIS, PAM, LDAP
Pre-WS Authentication/Authorization Credential Management
SimpleCA Simplified certificate authority
MyProxy Online credential repository for X.509 proxy
credentials
Globus Authentication
Globus Authentication
NIS
LDAP
PAM
Site A : Washington DC
Site B : UNT LDAP Student Login
Site C : Custom Authentication
User Authentication
Credential Repository
Context Services Info
Services
InfraServices
SecurityServices
Rsrc Mgmt Services
Execution Mgmt
Services
DataServices
PolicyMgmt
VOMgmt
Access
Integration
Transfer
Replication
BoundaryTraversal
Integrity
Authorization
Authentication
WSRF WSN WSDM
EventMgmt
Monitoring Discovery
JobMgmt
Logging
ExecutionPlanning
WorkflowMgmt
WorkloadMgmt
Provisioning
Execution
DeploymentConfigurationReservation
Naming
Self MgmtServices
HeterogeneityMgmt
Service LevelAttainment
QoSMgmt
Optimization
Information Services
Infrastructure Services
SelfMgmtServices
SecurityServices
Resource Mgmt Services
Execution Mgmt Services
DataServices
Context Services
CERN
CERN