Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELAE-infrastructure shared between Europe and Latin America gLite Overview Christian Grunfeld, UNLP 8th EELA Tutorial, La Plata, 11/12-12/12, 2006 E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 2 Middleware Grid Middleware Layer between user applications and grid resources. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 3 Grid Concepts VOs:Individuals and/or Institutions having direct access to resources. User Interface Grid services E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 4 Introduction Grid Sistems & Applications aim is to: Integrate Virtualise Manage Resources and services across different VOs. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 5 Grid Requirements Heterogeneous VO Resource Sharing Resource Utilization Job Execution Data Services Security Scalability Availability Specific Requeriments E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 6 Multi-VOs Users join VOs Virtual organisation contributes resources & negotiates access Grid middleware runs on each resource Storage elements Compute elements Additional services (both people and grid middleware) enable the grid Effect: collaboration INTERNE T E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 7 Building on GSI Build on Grid Security Infrastructure to create services that include: Job submission: run a job on a remote computer Information services: So I know which computer to use File transfer: so large data files can be transferred Replica management: so I can have multiple versions of a file close to the computers where I want to run jobs Production grids are (currently) based on the Globus Toolkit release 2 Globus Alliance: E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 8 gLite E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 9 gLite gLite is the next generation middleware for grid computing. Born from the collaborative efforts from academic and industrial research centers as part of the EGEE Project. The gLite Grid services follow a Service Oriented Architecture facilitate interoperability among Grid services allow easier compliance with upcoming standards Architecture is not bound to specific implementations services are expected to work together services can be deployed and used independently The gLite service decomposition has been largely influenced by the work performed in the LCG project E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 10 gLite Service Decomposition 5 High level services + CLI & API E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 11 gLite Security Services E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 12 gLite Security Services Authentication Identify entities (users, systems and services) when establishing context for message exchange (Who are you?). Aim - Provide a Credential having a universal value that works for many purposes across many infrastructures, communities, VOs and projects. gLite uses the PKI (X.509) infrastructure using CAs as thrusted third parties. MyProxy (http://grid.ncsa.uiuc.edu/myproxy/) Trust domain: The set of all EGEE CAs is our Trust Domain. Revocation: Identities must be revocated timely Credential Storage: Local or delegated credential (Services or Users) Privacy Preservation: Use of personal data E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 13 gLite Security Services Authorization Allows or denies access to services, based on policies. Agent: The user interacts with a centralized Authorization Server Push*: Authorization Services issue Tokens. Pull: The resource asks to the Authorization Services. Authorization Sources: Attribute Authority (AA): User Set of Attibutes. (VOMS) E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 14 gLite Grid Access Two possibilities: APIs and CLI. The use of web-services allows the automatic generation of APIs (error prone, lack of tools). E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 15 gLite Information and Monitoring Services Information services are vital low level component of Grids. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 16 gLite Information and Monitoring Services Basic info and monitoring services (RGMA) Information is provided by a Publish and Consume mechanism. Appearance of a single federated database to query through the SQL. Each VO has a VDB. Schema - Contains tables (GLUE) Registry List of available sources of information (Mediation) Producers Source of information (Primary, Secondary, On-demand) Consumers Make queries against tables (Continuous, Latest, History) E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 17 gLite Information and Monitoring Services Job Monitoring, Service Discovery, Network performance Monitoring Job Monitoring Java logging service, log4j, Apache/Chainsaw (for other languages). Service Discovery Locates suitable services to both users and services (Library!). Network Performance Monitoring Many network monitoring frameworks. Aim: perform a standard interface to those frameworks. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 18 gLite Job Management Services E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 19 gLite Job Management Services Accounting Accumulates information about the resource usage done by users or groups of users (VOs). Information on Grid Services/Resources needs sensors (Resource Metering, Metering Abstraction Layer, Usage Records). Records are collected by the Accounting System (Queries: Users, Groups, Resource) Grid services should register themselves with a pricing service when accounting for billing purposes. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 20 gLite Job Management Services Computing Element Service that represent the computing resource that is responsible of the job management: (submission, control, etc.) CEs refer to a set or a cluster of computational resources (WN) managed by LRMS, to dispatch jobs matching users requests. Two job submission models (accordingly to user requests and site policies) : PUSH(jobs pushed to CE queue), PULL(jobs coming from WMS when CE queue is empty) CE responsible to collect accounting information. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 21 gLite Job Management Services Workload Management WMS set of middleware components responsible of distribution and management of jobs across Grid resources. Two core components of WMS: WM: accept and satisfy requests for job management. Matchmaking is the process of assigning the best available resource. L&B: keep track of job execution in term of events: (Submitted, Running, Done,...) E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 22 gLite Job Management Services Job Provenance, Package Manager Job Provenance (JP) - Keeps track of submitted jobs for long periods (months, years). Package Manager Helper service to automate: installing, configuring, updating and removing of software components. (RPM, dpkg/APT, Portage, ) E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 23 gLite Data Services E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 24 gLite Data Services Storage Element Needed Service are at least: Storage back-end (Drivers and Hardware) SRM Interface (Storage Specific) Transfer service (GridFTP) Native POSIX like file I/O API (gLite-I/O) Auxiliary Accounting and Logging services E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 25 gLite Data Services Catalogs OS like file access metaphor. LFN (Logical file name) GUID (Grid unique identifier) SimLinks SURL (Site URL) TURL (Transfer URL) Catalogs: Authorization Base Metadata Base Metadata Schema Replica Catalog File Catalog File Authorization Metadata Combined Catalog Storage Index gLite - (FireMAN) E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 26 gLite Helper Services Configuration and Instrumentation Service Query service state. Agreement Service Implements a communication protocol for the SLAs. Bandwidth Allocation & Reservation service (BAR) Controlling, Balancing and Manage Network flows. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 27 Components Site: Computing Element (CE) Gateway to local computing resources (cluster de worker nodes) Worker Nodes (WN) Storage Element (SE) Gateway to local storage (disk, tape) A gridftp server, and SRM Interface, IO server User Interfaces (UI) Users access point to the grid Client programs using some/all grid services. E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 28 Components Grid or VO- wide Security Virtual Organization Server (VOMS) MyProxy server (Proxy) Information System Job handling Workload Management System (WMS) Logging & Bookkeping (LB) Data management File catalog (FiReMan) File Transfer Service (FTS) File Placement Service (FPS) E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 29 VOMS Virtual Organization Membership Service Multiple VOs Multiple roles in VO Compatible X509 extensions Signed by VOMS server Web admin interface Supports MyProxy Resources providers grant access to VOs or roles Sites map VO members/roles to local auth mechanism (unix users accounts) Allows for local policy E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 30 MyProxy Allows longer lived jobs / increases security WMS renews proxy Users should not produce long lives proxies Allows for secure user mobility Users does not need to copy globus-keys around E-infrastructure shared between Europe and Latin America Segundo Taller Latino Americano de Computacin GRID Primer Taller Latino Americano de EELA Primer Tutorial Latino Americano de EELA 31 References Globus Alliance:gLite middleware:gLite 3.0 user guide: https://edms.cern.ch/file/722398//gLite-3- UserGuide.htmlhttps://edms.cern.ch/file/722398//gLite-3- UserGuide.html