segundo taller latino americano de computación grid – primer taller latino americano de eela –...
TRANSCRIPT
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA
www.eu-eela.org
E-infrastructure shared between Europe and Latin America
gLite Overview
Grupo Grid
Universidad de Los Andes
Mérida, 24-29 April 2006
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 2
E-infrastructure shared between Europe and Latin America
• This presentation is based in previews presentations from:– Riccardo Bruno, – Salvatore Scifo– Mike Mineter
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 3
E-infrastructure shared between Europe and Latin America
Introduction
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 4
E-infrastructure shared between Europe and Latin America
Middleware
• Grid Middleware – Layer between user applications and grid resources.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 5
E-infrastructure shared between Europe and Latin America
Grid Concepts
• VOs:Individuals and/or Institutions having direct access to resources.
User Interface
User Interface
Grid services
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 6
E-infrastructure shared between Europe and Latin America
Introduction
• Grid Sistems & Applications aim is to:– Integrate– Virtualise– Manage
• Resources and services across different VOs.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 7
E-infrastructure shared between Europe and Latin America
Grid Requirements
• Heterogeneous• VO Resource Sharing• Resource Utilization• Job Execution• Data Services• Security• Administrative Costs• Scalability• Availability• Specific Requeriments
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 8
E-infrastructure shared between Europe and Latin America
Multi-VOs
• Users join VOs
• Virtual organisation contributes resources & negotiates access
• Grid middleware runs on each resource– “Storage elements”
– “Compute elements”
• Additional services (both people and grid middleware) enable the grid
• Effect:
collaboration
INTERNET
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 9
E-infrastructure shared between Europe and Latin America
Building on GSI
• Build on Grid Security Infrastructure to create services that include:– Job submission: run a job on a remote computer– Information services: So I know which computer to use– File transfer: so large data files can be transferred– Replica management: so I can have multiple versions of a file
“close” to the computers where I want to run jobs
• Production grids are (currently) based on the Globus Toolkit release 2 Globus Alliance: http://www.globus.org/
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 10
E-infrastructure shared between Europe and Latin America
Convergence of Web Services and Grids
Grid prototypesweb develo
pments
Web services
“big Science” research
INTERNET
World-wide web
High throughput-computing
Massively parallel computing
High-end computing
Open Grid Services Architecture
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 11
E-infrastructure shared between Europe and Latin America
gLite
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 12
E-infrastructure shared between Europe and Latin America
gLite
• gLite is the next generation middleware for grid computing.
• Born from the collaborative efforts from academic and industrial research centers as part of the EGEE Project.
• The gLite Grid services follow a Service Oriented Architecture– facilitate interoperability among Grid services – allow easier compliance with upcoming standards
• Architecture is not bound to specific implementations– services are expected to work together – services can be deployed and used independently
• The gLite service decomposition has been largely influenced by the work performed in the LCG project
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 13
E-infrastructure shared between Europe and Latin America
gLite - Middleware
• Many VOs need sharing of resources through services– Accessing– Allocating– Monitoring– Accounting
• gLite – Lightweight Middleware for Grid Computing
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 14
E-infrastructure shared between Europe and Latin America
gLite – Service Decomposition
5 High level services
+ CLI & API
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 15
E-infrastructure shared between Europe and Latin America
gLite – Security Services
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 16
E-infrastructure shared between Europe and Latin America
gLite – Security Services Authentication
Identify entities (users, systems and services) when establishing context for message exchange (Who are you?).
Aim - Provide a Credential having a universal value that works for many purposes across many infrastructures, communities, VOs and projects.
gLite uses the PKI (X.509) infrastructure using CAs as thrusted third parties.
MyProxy (http://grid.ncsa.uiuc.edu/myproxy/)
Trust domain: The set of all EGEE CAs is our Trust Domain.
•Revocation: Identities must be revocated timely
•Credential Storage: Local or delegated credential (Services or Users)
•Privacy Preservation: Use of personal data
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 17
E-infrastructure shared between Europe and Latin America
gLite – Security Services Authorization
Allows or denies access to services, based on policies.
•Agent: The user interacts with a centralized Authorization Server•Push*: Authorization Services issue Tokens.•Pull: The resource asks to the Authorization Services.
Authorization Sources:
•Attribute Authority (AA): User <-> Set of Attibutes. (VOMS)•Policy Assertions: Third party policies. (CAS)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 18
E-infrastructure shared between Europe and Latin America
gLite – Security Services
Auditing, Delegation, Sandboxing
Auditing - Monitoring and Post-Mortem analysis of security related events.
In computational grids It goes hand by hand with the accounting.
•Who did what?
•Where and when?
•In case of accounting:
•For how long?•For how much?
•Delegation: The need of delegate privileges to other entities is done by Proxy Certificates. This is the most widely adopted mechanism by Grid communities. (Also: Single Sign-On, Dynamic entity identification).
•Sandboxing - Grid applications need the isolation of assigned resources in a transparent fashion by Security services: AuthN and AuthZ. (Virtualisation).
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 19
E-infrastructure shared between Europe and Latin America
gLite – Grid Access
Two possibilities: APIs and CLI.
The use of web-services allows the automatic generation of APIs
(error prone, lack of tools).
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 20
E-infrastructure shared between Europe and Latin America
gLite – Information and Monitoring Services
Information services are vital low level component of Grids.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 21
E-infrastructure shared between Europe and Latin America
gLite – Information and Monitoring ServicesBasic info and monitoring services (RGMA)
•Information is provided by a Publish and Consume mechanism.•Appearance of a single federated database to query through the SQL.•Each VO has a VDB.
•Schema - Contains tables (GLUE)•Registry – List of available sources of information (Mediation)•Producers – Source of information (Primary, Secondary, On-demand)•Consumers – Make queries against tables (Continuous, Latest, History)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 22
E-infrastructure shared between Europe and Latin America
gLite – Information and Monitoring Services Job Monitoring, Service Discovery, Network performance Monitoring
•Job Monitoring – Java logging service, log4j, Apache/Chainsaw (for other languages).
•Service Discovery – Locates suitable services to both users and services (Library!).
•Network Performance Monitoring – Many network monitoring frameworks. Aim: perform a standard interface to those frameworks.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 23
E-infrastructure shared between Europe and Latin America
gLite – Job Management Services
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 24
E-infrastructure shared between Europe and Latin America
gLite – Job Management ServicesAccounting
•Accumulates information about the resource usage done by users or groups of users (VOs).
•Information on Grid Services/Resources needs sensors (Resource Metering, Metering Abstraction Layer, Usage Records).
•Records are collected by the Accounting System (Queries: Users, Groups, Resource)
•Grid services should register themselves with a pricing service when accounting for billing purposes.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 25
E-infrastructure shared between Europe and Latin America
gLite – Job Management ServicesComputing Element
•Service that represent the computing resource that is responsible of the job management: (submission, control, etc.)
•CEs refer to a set or a cluster of computational resources (WN) managed by LRMS, to dispatch jobs matching users requests.
•Two job submission models (accordingly to user requests and site policies):
• PUSH (jobs pushed to CE queue), • PULL (jobs coming from WMS when CE queue is empty)
•CE responsible to collect accounting information.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 26
E-infrastructure shared between Europe and Latin America
Computing Element (CE)
• Works in push or pull mode
• Site policy enforcement
• Exploit new Globus GK and Condor-C (close interaction with Globus and Condor team)
CEA … Computing Element Acceptance
JC … Job Controller
MON … Monitoring
LRMS … Local Resource Management System
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 27
E-infrastructure shared between Europe and Latin America
gLite – Job Management ServicesWorkload Management
•WMS set of middleware components responsible of distribution and management of jobs across Grid resources.
•Two core components of WMS:
•WM: accept and satisfy requests for job management.Matchmaking is the process of assigning the best available resource.
•L&B: keep track of job execution in term of events: (Submitted, Running, Done,...)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 28
E-infrastructure shared between Europe and Latin America
gLite – Job Management ServicesJob Provenance, Package Manager
•Job Provenance (JP) - Keeps track of submitted jobs for long periods (months, years).
•Package Manager – Helper service to automate: installing, configuring, updating and removing of software components. (RPM, dpkg/APT, Portage, …)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 29
E-infrastructure shared between Europe and Latin America
gLite – Data Services
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 30
E-infrastructure shared between Europe and Latin America
gLite – Data ServicesStorage Element
Needed Service are at least:
•Storage back-end (Drivers and Hardware)
•SRM Interface (Storage Specific)
•Transfer service (GridFTP)
•Native POSIX like file I/O API (gLite-I/O)
•Auxiliary Accounting and Logging services
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 31
E-infrastructure shared between Europe and Latin America
gLite – Data ServicesCatalogs
OS like file access metaphor.
•LFN (Logical file name)
•GUID (Grid unique identifier)
•SimLinks
•SURL (Site URL)
•TURL (Transfer URL)
Catalogs:
•Authorization Base•Metadata Base•Metadata Schema•Replica Catalog•File Catalog•File Authorization•Metadata•Combined Catalog•Storage Index
gLite - (FireMAN)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 32
E-infrastructure shared between Europe and Latin America
gLite – Data ServicesData Movement
•Data Scheduler (DS) Keep track of user/service transfer requests•File Transfer/Placement Sercice (FTS/FPS) •Transfer Queue (Table)•Transfer Agent (Network)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 33
E-infrastructure shared between Europe and Latin America
gLite – Helper Services
Configuration and Instrumentation Service – Query service state.
Agreement Service – Implements a communication protocol for the SLAs.
Bandwidth Allocation & Reservation service (BAR) – Controlling, Balancing and Manage Network flows.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 34
E-infrastructure shared between Europe and Latin America
Components
• Site:– Computing Element (CE)
Gateway to local computing resources (cluster de worker nodes)
– Worker Nodes (WN)– Storage Element (SE)
Gateway to local storage (disk, tape) A gridftp server, and SRM Interface, IO server
– User Interfaces (UI) User’s access point to the grid Client programs using some/all grid services.
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 35
E-infrastructure shared between Europe and Latin America
Components
• Grid – or VO- wide– Security
Virtual Organization Server (VOMS) MyProxy server (Proxy)
– Information System– Job handling
Workload Management System (WMS) Logging & Bookkeping (LB)
– Data management File catalog (FiReMan) File Transfer Service (FTS) File Placement Service (FPS)
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 36
E-infrastructure shared between Europe and Latin America
VOMS
• Virtual Organization Membership Service– Multiple VOs– Multiple roles in VO
Compatible X509 extensions Signed by VOMS server
– Web admin interface– Supports MyProxy– Resources providers grant access to VOs or roles– Sites map VO members/roles to local auth mechanism (unix
users accounts) Allows for local policy
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 37
E-infrastructure shared between Europe and Latin America
MyProxy
• MyProxy– Allows longer lived jobs / increases security
WMS renews proxy Users should not produce long lives proxies
– Allows for secure user mobility Users does not need to copy globus-keys around
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 38
E-infrastructure shared between Europe and Latin America
ReplicaReplicaCatalogueCatalogue
UIJDL
Logging &Logging &Book-keepingBook-keeping
ResourceResourceBrokerBroker
Job SubmissionJob SubmissionServiceService
StorageStorageElementElement
ComputeComputeElementElement
Information Information ServiceService
Job Status
DataSets info
Author.&Authen.
Job S
ub
mit
Even
t
Job
Qu
ery
Job
Stat
us
Input “sandbox”
Input “sandbox” + Broker InfoGlobus RSL
Output “sandbox”
Output “sandbox”
Job Status
Pu
blis
h
grid
-pro
xy-in
it
Exp
and
ed J
DL
SE & CE info
Segundo Taller Latino Americano de Computación GRID – Primer Taller Latino Americano de EELA – Primer Tutorial Latino Americano de EELA 39
E-infrastructure shared between Europe and Latin America
Questions