quadrics in linux clusters - sup.xenya.sisup.xenya.si/sup/info/quadrics/quadrics.pdf · a...
TRANSCRIPT
![Page 1: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/1.jpg)
A Finmeccanica Company
QUADRICS IN LINUX CLUSTERS
John Taylor
![Page 2: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/2.jpg)
A Finmeccanica Company
QLC 21/11/00• Motivation
• Quadrics Cluster Products
• Performance
• Case Studies
• Development Activities
![Page 3: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/3.jpg)
A Finmeccanica Company
Super-Cluster Performance Landscape
1 2 4 8 16 32 64 128 256 512 1024 2048 40961
2
4
8
16
32
64
128
Number of Nodes
CP
Us per S
MP
LINUX SUPERCLUSTERS
CLRC48 Gf
CPLANT~600 GF ?
![Page 4: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/4.jpg)
A Finmeccanica Company
Where is the HPC Market• HPC has migrated from MPP to:
– Clustered Shared Memory Systems• AlphaServer SC, IBM SP and Vector Machines
– LINUX Clusters (Alpha or x86)• “Commoditized Network”
– Quadrics solving the differential• degree of coherence/SSI - programming model,
manageability, administration
![Page 5: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/5.jpg)
A Finmeccanica Company
LINUX Pros and Cons• Open Source
• Wide Availability
• Early Availability
• Cheaper
• Leverage MPP S/W
• Support
• Security
• Scalability
• TPSW Availability
• Performance
![Page 6: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/6.jpg)
A Finmeccanica Company
Business StrategyTechnology Leadership in High PerformanceInterconnect and Cluster Management Software (QsNetand RMS)Technology and Business PartnershipsCreation of ChannelsHPC Services e.g Integration of v. large LINUX clusters
![Page 7: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/7.jpg)
A Finmeccanica Company
QSW extensible HPC ClusterComponents
• Generic Technology for tightly coupled clusters ofSMP’s
• Where tightly coupled means:– A hardware interconnect capable of scaling in both the number
of SMP’s and the number of CPU’s per node
– A “SSI” providing a coherent view of the system as a singleentity.
– The provision of application development environmentsconsistent with the DSM model
![Page 8: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/8.jpg)
A Finmeccanica Company
High Performance Cluster Productsfrom Quadrics
High performance interconnect
Resource management system
Parallel application developmenttools
Integrated TPSW Support
![Page 9: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/9.jpg)
A Finmeccanica Company
Architecture Overview• Management Node(s)
• Processing Nodes
• Quadrics Switch
• Management LAN
• Console Network
• Disk Array
![Page 10: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/10.jpg)
12/11/2000 Quadrics Supercomputers World LtdNDA Required
A Finmeccanica Company
Quadrics Interconnect (QsNet)• Two Custom Design ASICs make up the network
– Quadrics Network Adapter (elan)• 2nd Generation - 64bit/66 MHz. PCI -Based
• Very Low Latency , High Bandwidth
– QSW Multi-Stage Network (elite)• Modular Design , Fat Tree Topology
• Combined to provide high scalability, flexibility andtolerance
![Page 11: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/11.jpg)
A Finmeccanica Company
Network ComponentsNetwork Adapter
128 way Switch Chassis
16 way Switch Card
![Page 12: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/12.jpg)
12/11/2000 Quadrics Supercomputers World LtdNDA Required
A Finmeccanica Company
QsNet Adaptor• Intelligent PCI adapter
– DMA engine
– microprocessor w/64MB SDRAM
• One Sided Communications– Get/Put
– Send/Receive (TPORTS or Queued DMAs)
• OS Bypass with Virtual Addressing– no page locking or copying
– full protection
![Page 13: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/13.jpg)
A Finmeccanica Company
QsNet• Full bi-sectional
bandwidth
• Logarithmic cost
• Multiple routes
• Hardwarebroadcast
• General purposetopology
![Page 14: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/14.jpg)
A Finmeccanica Company
Quadrics 128-way SwitchFeatures
• 1-8 ×16 way switch cards
• 16 × 8 top switch cards
• Passive mid-plane
• 2 from 3 fault tolerant 48V PSU
• Live insertion
• Temperature, fan and PSU status
• Full JTAG boundary scan
• Performance– 42.5 Gbytes/sec bi-sectional
bandwidth
– 175 ns latency
![Page 15: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/15.jpg)
A Finmeccanica Company
Quadrics Cluster Software• Standard Software Hierarchy + Enhancements to
Couch Parallelism– Inter Processor Communication
– Single Point Cluster Management (Switch, Console)
– Scheduling of Parallel Programs
– Scaleable File System
– Accounting and Monitoring
– High Availability Strategies
– TPSW Support
![Page 16: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/16.jpg)
A Finmeccanica Company
Quadrics SoftwareRMS Products
Baseline (“RMS-lite”) free with hardwareValue Added Product
Operating SystemsTru64 UNIX V5.0 and V5.1Alpha Linux 2.2.14 and 2.4Intel Linux 2.4Solaris 2.6
![Page 17: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/17.jpg)
A Finmeccanica Company
Quadrics Software Components (1)• Baseline Product
• QsNet Linux drivers
• IP over QsNet
• MPI/SHMEM optimized for QsNet
• QsNet diagnostics
• Documentation (electronic)
![Page 18: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/18.jpg)
A Finmeccanica Company
Quadrics Software Components (2)• Full Product
– Single point installation and system management
– RMS parallel job scheduler
– Pandora Graphical User Interface
– Filesystem over QsNet
– TotalView support
– Documentation (electronic plus one paper copy)• The full product will be supplied as RPMs for the current product release
of RedHat Linux fully qualified on a range of platforms and licensed usingflexlm. Sources will be available to customers under a "no commercialreuse" license.
![Page 19: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/19.jpg)
A Finmeccanica Company
Software Overview• Kernel Services
![Page 20: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/20.jpg)
A Finmeccanica Company
Cluster Services• Node status monitoring
– bitmask of the functioning set of nodes
• Console logging
• Automated installation (Linux)
• Graphical User Interface – Pandora
![Page 21: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/21.jpg)
A Finmeccanica Company
Application Development• Standard compilers
• ATLAS Blas libraries
• MPI and Shmem
• Totalview
• VAMPIR (soon)
• PBS and LSF (future)
![Page 22: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/22.jpg)
A Finmeccanica Company
PerformanceStandard Benchmarks
Application Specific
![Page 23: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/23.jpg)
A Finmeccanica Company
Performance Overview• Line rate 400 Mbytes/s
• Peak data rate (adapter memory) 340 Mbytes/s
• MPI Send (33Mhz/64bit) 200 Mbytes/s
• DMA 2.5 µs
• MPI send 5 µs
• MPI Send (66MHz/64bit) 307 Mbytes/s
• DMA write 1.7 usec
• MPI send 4.5 usec
![Page 24: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/24.jpg)
A Finmeccanica Company
Network PerformanceRough Latency Budget
0
1
2
3
4
5
6
dping tping mpi mpi (66/64)
use
cs
mpi
tport
Elan
PCI
switch
cable
![Page 25: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/25.jpg)
A Finmeccanica Company
Quantum Simulations in Industry 25-27 September 2000
Computational Sc ience and Engine ering Department Daresbury Laboratory
Molecular Modelling on High-End andMolecular Modelling on High-End andCommodity-Type Computers:Commodity-Type Computers:
Status and PerspectivesStatus and Perspectives
MartynMartyn F. Guest and Paul Sherwood F. Guest and Paul Sherwood
CCLRC Daresbury LaboratoryCCLRC Daresbury Laboratory
[email protected]@daresburydaresbury.ac..ac.ukuk
http://www.cse .clrc.ac.uk/Activity/QUAS I
![Page 26: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/26.jpg)
A Finmeccanica Company
Development Activities• Porting to IA-32 and IA-64 Intel Systems
• Extending Current Generation network– Tracking the increase in Node “fatness”
– Increasing Node Count (Distributed Switch)
• Next generation network– Tracking the increase in CPU performance
– Fibre Interconnect - EMC is better.Copper is cheaper
• Software– High Availability Strategies
– High Performance File Systems
![Page 27: QUADRICS IN LINUX CLUSTERS - sup.xenya.sisup.xenya.si/sup/info/quadrics/Quadrics.pdf · A Finmeccanica Company Quadrics Software RMS Products Baseline (“RMS-lite”) free with hardware](https://reader033.vdocuments.us/reader033/viewer/2022050511/5f9b8bcf116916484a1b39dd/html5/thumbnails/27.jpg)
A Finmeccanica Company
References• Http://www.quadrics.com
• http://www.compaq.com/hpc• http://www.c3.lanl.gov/cic3/teams/par_arch/Publications
.html
• http://www.cse.clrc.ac.uk/Activity/QUASI
• http://www.psc.edu