the jülich research on petaflops architectures · jülich supercomputing centre juropa jülich...

The Jülich Research on PetaflopsArchitectures

Gilad Shainer, Mellanox Technologies

Dr. Thomas Lippert, Jülich Dr. Thomas Lippert, Jülich

Axel Koehler, Sun Microsystems

David Scott, Intel

Hugo R. Falter, ParTec

The Jülich Research on Petaflops Architectures Project

The JuRoPa II, one of the leading European PetaScale supercomputerprojects, is currently being constructed at the ForschungszentrumJülich, in the German state of North Rhine-Westphalia - one of thelargest interdisciplinary research centers in Europe. The new systemsare being built through an innovative alliance between Mellanox, Bull,Intel, Sun Microsystems, ParTec, and the Jülich SupercomputingCentre; the first such collaboration in the world. This new 'best-of-breed'

2

Centre; the first such collaboration in the world. This new 'best-of-breed'system, one of Europe’s most powerful, will support advanced researchin many areas such as health, information, environment, and energy. Itconsists of two closely coupled clusters, JuRoPA, with more than 200Teraflop/s performance, and HPC-FF, with more than 100 Teraflop/s.The latter will be dedicated for the European fusion researchcommunity. The session will introduce the project and the initiative, howit will effect future supercomputers systems, and how it will contribute toPetascale scalable software development

Dr. Thomas Lippert, Jülich

3

Mitg

lied

de

r H

elm

ho

ltz-G

em

ein

sch

aft

Thomas Lippert

Institute for Advanced Simulation

Jülich Supercomputing Centre

JuRoPAJuRoPAJülich Research on Petaflop/s Architectures

BoF

ISC09

June 23, 13:30-14:10

Jülich Supercomputing Centre (JSC)

523.6.2009 BoF JuRoPA / ISC09 Thomas Lippert, IAS/JSC/FZJ 5

IBM Blue Gene/LJUBL, 45 TFlop/s

2004

2005/6

IBM Power 4+JUMP, 9 TFlop/s

Developing Supercomputers @ JSC


2007/8IBM Blue Gene/PJUGENE, 223 TFlop/s

2009

File ServerGPFS

File ServerGPFS, Lustre

IBM Blue Gene/PJUGENE, 1 PFlop/s

Highly-Scalable

QPACE for QCD 100 TFQPACE for QCD 100 TF

IBM Power 6 JUMP, 9 TFlop/s

General-Purpose

Intel Nehalem Clusters

HPC-FF100 TFlop/s

JUROPA200 TFlop/s

JuRoPA + HPC-FF Cluster computer Bull NovaScale R422-E21080 nodes, 8640 cores 101 TF peak, Intel Nehalem24 GB memoryInfiniband QDR (Mellanox)ParaStation Cluster-OSHPC for Fusion


Cluster computer SUN-blades2208 nodes, 17664 cores 207 TF peak, Intel Nehalem48 GB memoryInfiniband QDR (SUN M9)ParaStation Cluster-OSGeneral Purpose HPC

JuRoPA II:Best-of-Breed Philosophy

The fastest and most energy efficient processor of its classThe fastest and most energy efficient processor of its class

The fastest scalable communication system with adaptive routingThe fastest scalable communication system with adaptive routing

The most scalable and stable open source cluster and MPI softwareThe most scalable and stable open source cluster and MPI software


The only parallel storage system with endThe only parallel storage system with end--toto--end data integrityend data integrity

The most flexible and tunable Linux kernelThe most flexible and tunable Linux kernel

A reliable and competent European integrator and cluster development A reliable and competent European integrator and cluster development

partnerpartner

High Speed Networks- the network is the computer

Mellanox QDR InfiniBand network

� High speed intra-application communication (MPI, etc.)

� I/O network

� QoS might be needed to support both


� QoS might be needed to support both

�� MPI through ParaStationMPI through ParaStation

Service network

� Basic node monitoring (temperature, fans, etc. (IPMI))

� Lights out management

� Connection to Constellation-rack's service processors

� Gigabit (or Fast?) Ethernet

Infiniband Topology


23 x 4 QNEM modules, 24 ports each

6 x M9 SUN switches, 648 ports max. each,468/276 links used

Mellanox MTS3600 switches (Shark), 36 ports, for service nodes

4 Compute Sets (CS) with 15 Compute Cells (CC) each

CC with 18 Compute Nodes (CN) and 1 Mellanox MTS3600 (Shark) switch each

Virtual 648-port switches constructedfrom 54x/44x Mellanox MTS3600

JuRoPAHPC-FF

Service Networks

Mellanox QDR InfiniBand network

� High speed intra-application communication (MPI, etc.)

� I/O network

� QoS might be needed for to support both


� QoS might be needed for to support both

Service network

� Basic node monitoring (temperature, fans, etc. (IPMI))

� Lights out management

� Connection to Constellation-rack's service processors

� Ethernet

Service Networks: Management

All higher level services

� DNS, LDAP, syslog

�� Monitoring (GridMon), batch, process management Monitoring (GridMon), batch, process management

through ParaStationthrough ParaStation


� System installation, (node-) software management

Higher level services partly distributed

� Circumvent possible scalability limitations

Hierarchical network

� Full bandwidth only within Constellation-rack

� 1 (or 2 bounded) up-links to central switch

� Service nodes directly connected to central switch

JuRoPA – Development Phase (2009-2010)

System is not expected to scale during first phase

� Besides Linpack

Start development project right after system runs

� Transform the cluster into a “real Supercomputer”


� Transform the cluster into a “real Supercomputer”

Aims

� End-to-end data integrity – Lustre with ZFS / SUN

� Lustre & HSM integration / Bull & CEA

� Mitigation of OS-jitter / ParTec & Novell

� Adaptive Routing / Mellanox

OS-jitter

Ideally a parallel-computer does just computation

Nodes run full-fledged OS

� Various daemons

� In-kernel services (FS)


� In-kernel services (FS)

� Interrupt handling

App. might be interrupted

Total loss per node small

� Low prob., short duration

Accum. loss might be huge

� # of nodes raises tot. prob.

Our approach

Development (Novell, SUN, Bull, Mellanox, ParTec)

Strip down OS to vital parts

� Skip almost all daemons, etc.

� Already done within the installation images


SLERT kernel

� This needs further work on Lustre

Gang-scheduling via prioritization (align noise)

� Re-prioritize on receiving of global signal (on IB)

� Remaining daemons only in particular (global) slots

� Also interrupt-handling in the same particular slots

Gilad Shainer, Mellanox Technologies

16

The Mellanox Ene-to-end 40Gb/s Effect

50%

100%

91.6%

17

0%

1GigE 10GigE 20G IB 40G IB

Mellanox Complete End-to-End Solutions

MTS3600

1U 36 port QSFP

BX4020 / 4010QDR to 10GbE and/or

2/4/8G FC Stateless Gateway

IS5XXX

108 to 648 ports

modular switch

Modular Switch Gateways

Fixed Switch Fabric Management

18

Accelerating 40Gb/s InfiniBand Deployment

1U 36 port QSFP

IS5030/5035

1U 36 port QSFP

10/20/40Gb/s InfiniBand 10GigE NICsAdapters

MTS3610

19U 324 port QSFP

Cables

Fabric Management

FabricIT

InfiniBand Technology Leadership

� Industry Standard• Hardware, software, cabling, management

• Design for clustering and storage interconnect

� Price and Performance• 40Gb/s node-to-node

• 120Gb/s switch-to-switch

• 1us application latency

• Most aggressive roadmap in the industry

� Reliable with congestion management

The InfiniBand Performance Gap

is Increasing

240Gb/s

(12X)

19

� Reliable with congestion management

� Efficient• RDMA and Transport Offload

• Kernel bypass

• CPU focuses on application processing

� Scalable for Petascale computing & beyond

� End-to-end quality of service

� Virtualization acceleration

� I/O consolidation Including storage

InfiniBand Delivers the Lowest Latency

Fibre Channel

Ethernet

60Gb/s

20Gb/s

120Gb/s

40Gb/s

80Gb/s (4X)

HPC Networking Solutions Required Capabilities

Highest BWLowest latency

Advanced

QoS

NetworkAdaptation

Congestion

Avoidance

Enhanced

Scalability

20

QoS Avoidance

Hybrid Topologies

Applications

OffloadsHardware

Reliability

For PetaScale and Beyond

� Highest performance - highest throughout, lowest latency

� Network Adaptation- ensures highest efficiency

� Self recovery - ensures highest reliability

� Scalability - the solution for Peta/Exa flops systems

� On-demand resources - allocation per demand

21

� On-demand resources - allocation per demand

� Green HPC - lowering system power consumption

Proud Member of the HPC Advisory Council

� WW HPC organization (over 80 members)

• Bridge the gap between HPC usage and its potential

• Provide best practices and a support/development center

• Explore future technologies and future developments

• Explore advanced topics – HPC in a cloud, HPCaaS etc.

� For more info: http://www.hpcadvisorycouncil.com

22

Cluster Center

Axel Koehler, Sun Microsystems

23

The Sun Open Petaflop Architecture

Axel KöhlerSenior HPC Architect

Sun Microsystems GmbH

Sun Constellation System Open Petascale Architecture

NetworkingCompute Storage Software

Developer Tools

Provisioning

Grid Engine

Ultra-DenseBlade Platform

� Fastest processors: AMD Opteron, Intel Xeon, SPARC CMT

� Highest compute density

� Fastest host channel adaptor

Ultra-Denseand Ultra-Slim Switch Solutions

� 72, 648 and 3456 port InfiniBand switches

� Unrivaled cable simplification

� Most economicalInfiniBand cost/port

Comprehensive Software Stack

� Integrated developer tools

� Integrated Grid Engine infrastructure

� Provisioning, monitoring, patching

� Simplified inventory management

Linux

Ultra-DenseStorage Solution

� Most economicaland scalable parallelfile system building block

� Up to 48 TB in 4RU

� Up to 2TB of SSD

� Direct cabling toIB switch

The Sun Constellation Rack

• Density optimized blade rack> Redundant power and cooling

• 96 or 192 sockets per rack> Currently up to 9 TFLOPS Peak

• Optional cooling doors• Optional cooling doors> 30+ KW heat

• Fabric support> Infiniband, ...

• Chassis Monitoring Modules

• Designed for future CPU, memory and I/O technologies

Sun Blade X6275

Sun Infiniband QDR products

M9: Up to 648 Quad Data Rate ports QDR NEM: 8 iPASS connectors with 2 fully meshed internal InfiniScale IV switches

4x QDR IB ports to 24 Server Nodes

DR

12x 12x 12x

36 Port QDR IB Switch

InfiniScale IV

12x 12x 12x12x 12x

36 Port QDR IB Switch

InfiniScale IV

12 links

internal

12 12x QDR IB ports to core switch or next neighbour

Sun Project “M9”

• Modularity> Up to 3 per rack

> 9 horizontal line cards

> 9 vertical fabric cards

> 2 redundant chassis management controllers

• Line Card …9

lin

e

card

s…

…9 fabric cards…

1-8 VLs per Path

Multiple

Paths

Designed to maximize density and minimize cabling burdens with CXP

12x 3:1 total cable reduction

• Line Card> 4 Mellanox InfiniScale 4 chips

> 24 CXP connectors (72 ports QDR)

> Hot swap

• Fabric Card> 2 Mellanox InfiniScale 4 chips

> No external connectors

> Hot swap

…

Project “M9” configuration

Ports 648 4x QDR/DDR/SDR

Bisection Bandwidth 6,480 Tb/s

Latency 300 nS (QDR)

Physical size (WxHxD) 17.5”x19”x27”

Line Cards 9

Fabric Cards 9

4x Ports/LC 72

Switch chips/LC 4Switch chips/LC 4

Switch chips/FC 2

Switch chips 54

Power N+1

PSUs 4 Hot Swap

Power Consumption 7kW

Cooling Front to rear, Redundant

Fan Modules 36 Hot Swap

Chassis Management (CM) IPMI / ATCA

CM Controllers Redundant

Temperature 5-35degC

Altitude 0-3000m

Humidity 5-90% non-condensing

Connection from one QNEM With 8 M9 switches up to 5184 nodes

Serv

er

Nodes (

12 V

ayu B

lades )

12x

12x

12x

12x

36 P

ort

QD

R IB

Sw

itch

InfiniS

cale

IV

4x Q

DR

IB

port

s to 2

4 S

erv

er

Nodes (

DR

12x

12x

12x

36 P

ort

QD

R IB

Sw

itch

InfiniS

cale

IV

12x

36

12

lin

ks

inte

rna

l

One 12x cable connecting 3 server with the switch

Cables and connectors

• 12x QDR is CXP

> CXP is an Industry Standard

> 3:1 Cable reduction

> Optical available in 10 & 20M

> Copper in 1M, 2M, 3M, and 5M> Copper in 1M, 2M, 3M, and 5M

• QDR Splitter Cables

> 1 CXP to 3 QSFP

12x CXP to CXP

Copper

12x Optical Splitter

1CXP to 3 QSFP

12x CXP to CXP

Optical

JuRoPA Cluster 2208 Compute Nodes with

� 17664 Cores

� 52.99 TB DDR3 memory

� 207 TFLOPS Peak

2x2 Compute Blades (23 SB6048 Racks )

2208 Dual Socket QC Nehalem EP @2.93 GHz

w. 24GB DDR3 Memory (12x2GB DDR3) + Flash Card

6 x M9 Magnum QDR

with each 6 CXP LC (2592 Ports)

2 x virtual Magnum QDR

based on Mellanox 36 Port Switches

500 + 300 TB Capacity

Conclusion

• Sun Constellation is an open and scalable platform

• Scalable QDR Infiniband solution (M9 switches, QNEM, 12x Cables)

• Scalable storage• Scalable storage> Lustre parallel filesystem

• Open Source software stack

• Infrastructure efficient solutions

The Sun Open Petaflop The Sun Open Petaflop Architecture

Axel Kö[email protected]

David Scott, Intel

36

Intel® Xeon® Processor5500 Series (Nehalem-EP)

Architecture and Performance

3737

Dr. David S. Scott

Petascale Product Line Architect

June 2009

Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products asmeasured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other

sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and onthe performance of Intel products, visit http://www.intel.com/performance/resources/benchmark_limitations.htm or call (U.S.) 1-800-628-8686 or 1-916-356-3104.

Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customersto visit the referenced Web sites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate andreflect performance of systems available for purchase.

Relative performance is calculated by assigning a baseline value of 1.0 to one benchmark result, and then dividing the actual benchmark result for the baselineplatform into each of the specific benchmark results of each of the other platforms, and assigning them a relative performance number that correlates with theperformance improvements reported.

SPEC, SPECint, SPECfp, SPECrate. SPECpower, SPECjAppServer, SPECjbb, SPECjvm, SPECWeb, SPECompM, SPECompL, SPEC MPI, are trademarks of

the Standard Performance Evaluation Corporation. See http://www.spec.org for more information. TPC-C, TPC-H, TPC-E are trademarks of the TransactionProcessing Council. See http://www.tpc.org for more information.

Intel® Virtualization Technology requires a computer system with an enabled Intel® processor, BIOS, virtual machine monitor (VMM) and, for some uses, certainplatform software enabled for it. Functionality, performance or other benefits will vary depending on hardware and software configurations and may require a BIOS

update. Software applications may not be compatible with all operating systems. Please check with your application vendor.

Legal Disclaimers

3838

update. Software applications may not be compatible with all operating systems. Please check with your application vendor.

Hyper-Threading Technology requires a computer system with a processor supporting HT Technology and an HT Technology-enabled chipset, BIOS and operatingsystem. Performance will vary depending on the specific hardware and software you use. For more information including details on which processors support HTTechnology, see here

Intel® Turbo Boost Technology requires a Platform with a processor with Intel Turbo Boost Technology capability. Intel Turbo Boost Technology performancevaries depending on hardware, software and overall system configuration. Check with your platform manufacturer on whether your system delivers Intel TurboBoost Technology. For more information, see http://www.intel.com/technology/turboboost ”

Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor series, not across different processorsequences. See http://www.intel.com/products/processor_number for details. Intel products are not intended for use in medical, life saving, life sustaining, criticalcontrol or safety systems, or in nuclear facility applications. All dates and products specified are for planning purposes only and are subject to change without notice

* Other names and brands may be claimed as the property of others.

Copyright © 2009 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon and Intel Core are trademarks or registered trademarks of Intel Corporation or itssubsidiaries in the United States and other countries. All dates and products specified are for planning purposes only and are subject to change without notice

• Intelligent Performance

• Adaptable Energy Efficiency

• Flexible Virtualization

Enabled by

– Intel® Microarchitecture Nehalem

– 45 nm Hi-K Quad Core processor

Intel® Xeon® 5500 Processor

3939

– 45 nm Hi-K Quad Core processor

– Intel® Turbo Boost Technology

– Intel® Hyper-Threading Technology

– Enhanced Virtualization Technology

– Integrated Power Gates

– Automated Low-Power States

– Intel® Node Manager

A new generation of Intelligent Server Processors

� New Memory Subsystem

� Intel® QuickPath Interconnect

� Intel® Intelligent Power Technology

� New I/O Subsystem

Intel® 5520

Chipset

NEW!

NEW!Intel® Data

Center Manager

Intel® Node Manager

NEW!

Intel® Xeon® 5500 Platform

4040

� New I/O SubsystemPCI Express* 2.0

ICH 9/10Intel® X25-E

SSDs

NEW!

Intel® Node Manager

Massive Platform level innovations

Intel® Xeon® Processor 5500 series based Server platformsHPC Performance comparison to Xeon 5400 Series

Relative Performance Higher is better

4141

Exceptional gains on HPC applications

Energy Open MP Energy Open MP Weather EnergyWeather FEA FEA CFD CFD

Xeon 5400

series

Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or

software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on

performance tests and on the performance of Intel products, visit http://www.intel.com/performance/resources/limits.htm Copyright © 2009, Intel Corporation. * Other names and brands may be claimed as the property of others.

Source: Published/submitted/approved results March 30, 2009. See backup for additional details

Intel® Xeon® Processor 5500 series based Server platformsSPEC MPI* 2007 performance on a multi-node cluster

Higher is better

4242Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or



Source: Published/submitted/approved results March 30, 2009. See backup for additional details

• Key DetailsCluster configuration, Comparison on equal number of Nodes, Each node contains two quad-core cpus. 1 thread/core

Result published by SGI on SGI Altix ICE 8200EX* server based cluster, Comparison based on “base” score results

All data based on published results at http://www.spec.org/mpi2007/results/mpi2007.html as of March 30, 2009

• Benchmark notesBenchmark suite for evaluating MPI-parallel, floating point, compute intensive performance for cluster and SMP hardware.

Developed from native Message Passing Interface (MPI) parallel end-user applications

Excellent cluster scaling on SPEC MPI 2007

• Maximum B/W:

– DDR3 1333 across 3 channels

– Up to 1 DPC (6 DIMMs total)

– Max capacity: 48 GB

• General purpose:



– Max capacity: 96GB

CPU CPU

CPU CPU

10.6 GB/s

10.6

10.6

8.5 GB/s

8.5

8.5

CPUs

E5520and above

X5550and above

27208

33203

36588Higher is better

Stream Bandwidth Stream Bandwidth –– Mbytes/Sec (Triad)Mbytes/Sec (Triad)

+274%

1333 MHz memory

1066 MHz memory

800 MHz

Intel® Xeon® Processor 5500 series based Server platformsStream Bandwidth for Xeon X5570 processor

4343

• Maximum capacity:



– Max capacity: 144GB

(DPC – Dimms Per Channel)

CPU CPU

6.4 GB/s

6.4

6.4

All

SKUs

6102

9776

HTN 3.16/ BF1333/ 667

MHz mem

HTN 3.00/ SB1600/ 800

MHz mem

NHM 2.93/ 800 MHz mem/3

DPC

NHM 2.93/ 1066 MHz

mem/2 DPC

NHM 2.93/ 1333 MHz

mem/1 DPC

Massive Increase in Platform Bandwidth

Nehalem-EP memory Bandwidth for different configuration

Memory speed 800 MHz 1066 MHz 1333 MHz

1 DPC 2 DPC 3 DPC 1 DPC 2 DPC 1 DPC

Stream Triad 27748 26565 27208 33723 33203 36588

800 MHz memory

Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or



Source: Intel internal measurement – March 2009

Hugo R. Falter, ParTec

44

ParaStationV5

Operating and Management Software

BoF / The Jülich Research on Petaflops Architectures 45

Operating and Management Software

Hugo R. Falter, COO

ParaStationV5

ParaStation SoftwareBorn at the University of Karlsruhe and further developed

by ParaStation-Consortium with Forschungszentrum Jülich and ParTec

Designed for stability, robustness and scalability

in productive environments


Includes a lot of modules

(to mention only a few):ParaStationMPI

Process Management

System ProvisioningGridMonitor

Support & Services

ParTec‘s contribution to JuRoPA-JSC and HPC-FF

Development of ParaStationV5 Software family

HPC consulting, planning and cluster configuration

HPC software installation and benchmarking


HPC software installation and benchmarking

Software maintenance services & bug fixing

On-site & remote cluster management

Application integration

Co-Development

Goal Development of next generation of general purpose cluster as a research activity

Bring together the most experienced companies and research site to combine and integrate the best processor,

server, interconnect and cluster software


To overcome the high scalability limitations of today’s technology

Making one of Europe’s most powerful supercomputer available(274,8 Teraflops @ 91,6% efficiency with ParaStationMPI,

supporting more then 25,000 MPI tasks)

Contributions of all partners in the spirit of common interests

Outlook

Make outcome to new cluster projects available

Enhancing Cluster Software to a high level of stability and productivity

Demonstrating high scalability on interconnect and communication software


Providing comprehensive support services for the satisfaction

the customer

Duplicating experiences on smaller cluster systems

Bringing sustainability of new technology to the market

the jülich research on petaflops architectures · jülich supercomputing centre juropa jülich...

Documents