grids, clouds and hpc - the munich network management team
TRANSCRIPT
Grids, Clouds and HPC -The Munich Network Management Team
Dieter Kranzlmü[email protected]
Lehr- und Forschungseinheit für Kommunikationssysteme und Systemprogrammierung
D. Kranzlmüller Visit @ University of Virginia 2
About us - The MNM-Team.
D. Kranzlmüller Visit @ University of Virginia 3
Prof. Dr. Dieter Kranzlmüller, Prof. Dr. Heinz-Gerd Hegering
Dr. Victor ApostolescuDr. Michael Brenner
Dr. Ernst BötschDr. Wolfgang Hommel
Patricia MarcuDr. Helmut ReiserChristian Richter
Dr. Thomas SchaafDr. David Schmitz
Dr. Mark Yampolskiy
Prof.Dr. Gabi Dreo RodosekVolker Eiseler
Frank EyermannSebastian HanigkIris Hochstatter
Robert KochMichael Kretzschmar
Björn Stelte
Members of the MNM Team
Dr. Vitalian DanciuDr. Nils gentschen Felde
Christof KlauseckerSilvia Knittl
Annette KostelezkyThomas Köckerbauer
Ralf KönigTobias Lindinger
Feng LiuMartin Metzker
Dr. Michael SchiffersChristoph Spielman
Johannes Watzl
D. Kranzlmüller Visit @ University of Virginia 4
Topics
Service Management
Networks Grid computing
IT Security
HighPerformance
Computing
Virtualization
Cloud Computing
D. Kranzlmüller Visit @ University of Virginia 5
MNM-Team Lectures
D. Kranzlmüller Visit @ University of Virginia
Munich Region: High-Tech Cluster…
Information andCommunication
•Siemens•Infineon
•Sun•Intel•HP
•General Electric•…
Software•SoftLab
•Nemetschek•Oracle
•Microsoft•sd&m
Aerospace•EADS•ESG
•Galileo Industry•Astrium•IABG
•Eurocopter• …
Life Sciences, Media,
Telecommunication,Environmental Engineering…
Finance•Allianz
•Bay. Landesbank•HypoVereinsbank•MünchnerRück
• …
Venture Capitalists•3i
•Apax•Atlas Venture
•EarlyBird•TVM
•Wellington Partners• …
Research•DLR•FhG
•Max-Planck•LMU•FHM•TUM
•Univ. der Bundeswehr
• …
Patent•European Patent
Office•German Patent
Office
Automotive•BMW•Audi•MAN• …
6
Leibniz Supercomputing Centreof the Bavarian Academy
of Sciences and Humanities
© 2009 Leibniz Supercomputing Centre
By Ernst A. Graf
7
D. Kranzlmüller Visit @ University of Virginia
The Leibniz Supercomputing Centre•Computer Centre (~175 employees) for all Munich Universities with
-more than 80,000 students and-more than 26,000 employees-including 8,500 scientists
•Regional Computer Centre for all Bavarian Universities-Capacity computing-Special equipment-Backup and Archiving Centre (more than 7 petabyte, 5.5 billion files)
-Distributed File Systems-Competence centre (Networks, HPC, IT Management)
•National Supercomputing Centre-Gauss Centre for Supercomputing-Integrated in European HPC and Grid projects(DEISA, PRACE, EGI)
8
D. Kranzlmüller Visit @ University of Virginia
LRZ Supercomputer HLRB2
9
Foto Helmut Payer, produced by gsiCom
D. Kranzlmüller Visit @ University of Virginia
The LRZ: a Supercomputing Centre• National supercomputing system SGI Altix 4700
-9728 compute cores (Intel Itanium2 Montecito)-62.3 TFlop/s peak performance-56.5 TFlop/s Linpack benchmark-39 TByte Total Memory-660 TByte attached Disk space-Weighing 103 metric tons -Consuming ~1000 kVA-On 24 m x 12 m footprint
• 128 cpu SGI Altix 3700 Bx2 for scientists in the state of Bavaria
• 128 dual core cpu SGI Altix 4700 for scientists in the state of Bavaria
• Linux cluster with more than 3.500 cpus (resp. cores) mainly for the Munich universities and for scientists in the state of Bavaria
• More than 500 additional servers for general IT services
Foto Helmut Payer, produced by gsiCom
10
D. Kranzlmüller Visit @ University of Virginia
Examples of Applications @ LRZ•Computational Fluid Dynamics: Optimisation of turbines and wings,
noise reduction, air conditioning in trains•Fusion: Plasma in a future fusion reactor (ITER)•Astrophysics: Origin and evolution of stars and galaxies•Solid State Physics: Superconductivity, surface properties•Geophysics: Earth quake scenarios•Material Science: Semiconductors•Chemistry: Catalytic reactions•Medicine and Medical Engineering: Blood flow, aneurysms, air
conditioning of operating theatres•Biophysics: Properties of viruses, genome analysis•Climate research: Currents in oceans
11
D. Kranzlmüller Visit @ University of Virginia
Distribution of CPU-hours
12
D. Kranzlmüller Visit @ University of Virginia
Extension of the Buildings(Computer: March 2011, Staff: Autumn 2011)
Fotomontage
Südansicht
Nordansicht
13
D. Kranzlmüller Visit @ University of Virginia 14
Topics
Service Management
Grid computing
HighPerformance
ComputingCloud Computing
Networks
IT SecurityVirtualization
Grid computing
D. Kranzlmüller Visit @ University of Virginia25th NORDUnet
Conference, Copenhagen
15
290 Sites55 Countries144.000 Cores25 PetaBytes Disk>17.000 Users>200 VOs>330.000 Jobs/Day
ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…
Status July 2009: http://project.eu-egee.org/index.php?id=417
Jobs/month: +45% in a yearSites: +5% in a year
Countries: +10% in a yearVOs: +29% in a year
D. Kranzlmüller Visit @ University of Virginia
European Grid InitiativeObjectives:•Ensure the long-term sustainability of the European e-infrastructure
•Coordinate the integration and interaction between National Grid Infrastructures
•Operate the European level of the production Grid infrastructure for a wide range of scientific disciplines to link National Grid Infrastructures
EGI Grid Infrastructure should be•a large-scale, production Grid infrastructure •built on national grids that interoperate seamlessly at many levels,
•offering reliable and predictable services to a wide range of applications
D. Kranzlmüller Visit @ University of Virginia
EGI Operations TasksOperation of tools and services• Grid configuration repositories• Grid accounting repositories• Grid repositories for SLA
compliance and performance monitoring
• Grid operations portal• NGI Grid oversight
Security• Security policy development and
maintenance• Coordination of security and
incident response• Expert team for security
vulnerabilities
User support• Central ticket handling system• Gathering requirements for user
support tools
Other international tasks• MW deployment/roll-out and
support• Resource allocation & brokering
support• Interoperations between NGI’s and
with other grids• Network support• Definition of best practises,
procedures, requirements• Catch-all production grid core
services
EGI Blueprint Proposal (V3.0)http://www.eu-egi.eu/blueprint.pdf
• Functions of EGI• Financing of EGI• Transition to EGI
D. Kranzlmüller Visit @ University of Virginia
EGI Operations
EGI.eu EGI.eu global tasksNGI international tasksNGI local tasks
D. Kranzlmüller Visit @ University of Virginia
EGI Infrastructure
…
International Scientific and Research CollaborationInternational Scientific and Research Collaboration
NationalGrid
Initiative 1
NationalGrid
Initiative 1
NationalGrid
Initiative 2
NationalGrid
Initiative 2
NationalGrid
Initiative N
NationalGrid
Initiative N
Middlew
are A
Middlew
are B
Middlew
are B
Middlew
are X
gLite
UNICO
RE
ARC
EGI.euEGI.eu European-level Grid Services
Middleware… is essential for the grid… continues to evolve with
new functionalities… development is a
long-term process
D. Kranzlmüller Visit @ University of Virginia
D-Grid Project D-MON
ComputingService (DB)
VO-specific View (OGSA-DAI)
D. Kranzlmüller Visit @ University of Virginia 21
Topics
Service Management
HighPerformance
Computing
Networks Grid computing
IT SecurityVirtualization
Cloud Computing
Cloud Computing
D. Kranzlmüller Visit @ University of Virginia
Cloud Research @ MNM-Team
Questions:•What is the cloud?•What are the differencesbetween clouds and grids?
•Which technologies can beused within a cloud?
•How to use anonymousservices of cloudsefficiently?
•How to securely deploy and use clouds?
22
D. Kranzlmüller Visit @ University of Virginia 23
Cloud Testbed @ MNM
•Example: Zimory-T-Labs spin-off-Small company (~20 staff workers)-Interconnects 2 Telekom computing centers-Only enterprise-grade customers-Provides qualitatively high services-3 service-levels (Gold, Silver, Bronze)
D. Kranzlmüller Visit @ University of Virginia 24
Management solution to manage multiple clouds
Source: Zimory GmbH
Zimory Cloud (1/3)
D. Kranzlmüller Visit @ University of Virginia 25
•Gateway to enterpise cloud computing Resources•Support of SLAs
Source: Zimory GmbH
Zimory Cloud (2/3)
D. Kranzlmüller Visit @ University of Virginia 26Source: Zimory GmbH
Cloud GatewayDate Center manager, Orchestration layer, Security manager, Data policy manager, Resource quality managerData Center 1 Data Center 2
Cloud HostSecurity management, Resource
optimization, Interoperable access management, VM
migration, Virtualized Resources Virtualized Resources
VMware, Xen, KVM, Hyper-V VMware, Xen, KVM, Hyper-V
DB DB DB
Data Storage
DB DB DB
Data Storage
Database Hypervisor™Data management, Data
synchronization, Data migration, Storage reliability and security
Oracle, MySQL,… SAN Oracle, MySQL, … SAN
Cloud ManagerSLA manager, Resource planner,
Application profiler, Network optimizer, Load balancer, Migration manager, Accounting, Authentication, Policy
manager, Open APIInfrastructure ManagerInfrastructure Manager
vCenter, Platespin, etc. vCenter, Platespin, etc.
Zimory Public Cloud Solution
Wide Area Network
Zimory Cloud (3/3)
D. Kranzlmüller Visit @ University of Virginia 27
Topics
Service Management
Grid computing
HighPerformance
ComputingCloud Computing
Networks
IT SecurityVirtualization
HighPerformance
Computing
D. Kranzlmüller Visit @ University of Virginia
Scalable Parallel Debugging with
•Integrated workbench framework to access the power of existing computing infrastructures
•Built on top of the Eclipse framework, released under Eclipse Public License (EPL), continues as an Eclipse Technology Project
•Version 1.0 available for downloadhttp://www.eclipse.org/geclipse/
28
D. Kranzlmüller Visit @ University of Virginia 29
g-Eclipse Trace Viewer•Tool to visualize and analyze communication of parallel message passing programs
•Parallel debugging and development activities centered around Trace Viewer
•Integrated into g-Eclipse•Can be used asstandalone application
•Platform independent
D. Kranzlmüller Visit @ University of Virginia 30
g-Eclipse Trace Viewer
logical clocks
physical clocks
statistics
D. Kranzlmüller Visit @ University of Virginia
Scalability for Debugging•Abstraction techniques are necessary•Based on Actions and Markers extension points
-reorder, group or hide processes-mark interesting events
31
D. Kranzlmüller Visit @ University of Virginia
•By manually selecting the processes•Automatically by criteria
Reorder, Group or Hide Processes
32
D. Kranzlmüller Visit @ University of Virginia
•Search for patterns defined using a description language-serves program understanding
-helps to detect errors and inefficient communication
33
Communication Patterns
D. Kranzlmüller Visit @ University of Virginia 34
g-Eclipse Components Combined
D. Kranzlmüller Visit @ University of Virginia 35
Topics
Service Management
Networks Grid computing
IT Security
HighPerformance
Computing
Virtualization
Cloud Computing