overview of xsede and ut’s hpc resources · 2013. 10. 15. · • jülich supercomputing centre...

Overview of XSEDE and UT’s HPC resources

Daniel Lucio

Tuesday, October 15, 13

What is XSEDE?

• The Extreme Science and Engineering Discovery Environment (XSEDE) is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world.

• It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise.

• It’s a five-year, $121-million project supported by the National Science Foundation that funds ~140FTEs across 18 partner institutions. It replaced and expanded on the past NSF TeraGrid project.


What is XSEDE?

• XSEDE supports 8 supercomputers and high-end visualization and data analysis resources across the country. XSEDE integrates these resources and services, makes them easier to use, and helps more people use them.

• XSEDE is a socio-technical ecosystem

• Highly distributed organization: a project that involves staff at 18 partner institutions

• A completely virtual organization: breaking new ground from an organizational structure and management point of view

• Highly distributed engineering project: developing new methodologies to adapt traditional practices to the unusual context of XSEDE.


The partnership includes

• University of Illinois' National Center for Supercomputing Applications• Cornell University Center for Advanced Computing• Indiana University• Jülich Supercomputing Centre• Michigan State University• National Center for Atmospheric Research• National Center for Supercomputing Applications - University

of Illinois at Urbana-Champaign• National Institute for Computational Sciences - University of

Tennessee Knoxville/Oak Ridge National Laboratory• Ohio Supercomputer Center - The Ohio State University• Pittsburgh Supercomputing Center - Carnegie Mellon

University/University of Pittsburgh

• Purdue University• Rice University• San Diego Supercomputer Center - University of California

San Diego• Shodor Education Foundation• Southeastern Universities Research Association• Texas Advanced Computing Center - The University of Texas

at Austin• University of California Berkeley• University of Chicago• University of Southern California• University of Virginia


http://www.cac.cornell.edu/

http://www.cac.cornell.edu/

http://www.indiana.edu/

http://www.indiana.edu/

http://www2.fz-juelich.de/jsc/

http://www2.fz-juelich.de/jsc/

http://www.msu.edu/

http://www.msu.edu/

http://ncar.ucar.edu/

http://ncar.ucar.edu/

http://www.ncsa.illinois.edu/

http://www.ncsa.illinois.edu/

http://illinois.edu/




http://www.nics.tennessee.edu/

http://www.nics.tennessee.edu/

http://www.utk.edu/

http://www.utk.edu/

http://www.utk.edu/

http://www.utk.edu/

http://www.ornl.gov/

http://www.ornl.gov/

http://www.osc.edu/

http://www.osc.edu/

http://www.osu.edu/

http://www.osu.edu/

http://www.psc.edu/

http://www.psc.edu/

http://www.cmu.edu/index.shtml




http://www.pitt.edu/

http://www.pitt.edu/

http://www.purdue.edu/

http://www.purdue.edu/

http://www.rice.edu/

http://www.rice.edu/

http://www.sdsc.edu/

http://www.sdsc.edu/

http://www.ucsd.edu/




http://www.shodor.org/

http://www.shodor.org/

http://www.sura.org/home/index.html

http://www.sura.org/home/index.html

http://www.tacc.utexas.edu/

http://www.tacc.utexas.edu/

http://www.utexas.edu/




http://berkeley.edu/

http://berkeley.edu/

http://www.uchicago.edu/index.shtml

http://www.uchicago.edu/index.shtml

http://www.usc.edu/

http://www.usc.edu/

http://www.virginia.edu/

http://www.virginia.edu/

Available HPC resources

• Stampede debuted Intel's new innovative MIC technology on a massive scale, it commenced production in January, 2013.

• Blacklight is a SGI shared memory system intended for applications that require a large shared memory for computational tasks.

• Gordon is a unique, a flash-based supercomputer designed for data-intensive applications.

• Keeneland is a balanced hybrid CPU/GPGPU system for use with codes that can take advantage of accelerator performance.


Available HPC resources

• Kraken is a Cray XT5 system with compute nodes interconnected with SeaStar, a 3D torus. It is intended for highly scalable parallel applications.

• Lonestar is a Dell Linux Cluster, is a powerful, multi-use cyberinfrastructure HPC and remote visualization resource.

• Mason is a large memory computer cluster configured to support data-intensive, high-performance computing tasks using genome assembly software.

• Trestles employs flash-based memory and is designed for modest-scale research providing very fast turnaround time.


Other HPC resources

• Longhorn is a large visualization cluster designed for remote interactive visualization and data analysis.

Visualization

• Open Science Grid is a multi-disciplinary partnership to federate local, regional, community and national cyberinfrastructures to meet the researchers' high throughput computing needs.

High Throughput Computing

• Data Oasis (SDSC), Data SuperCell (PSC), HPSS (NICS), Ranch (TACC), XWFS (XSEDE)

Storage


http://www.xsede.org/sdsc-dataoasis

http://www.xsede.org/sdsc-dataoasis

http://www.xsede.org/psc-data-supercell

http://www.xsede.org/psc-data-supercell

http://www.xsede.org/nics-hpss

http://www.xsede.org/nics-hpss

http://www.xsede.org/tacc-ranch

http://www.xsede.org/tacc-ranch

http://www.xsede.org/xwfs

http://www.xsede.org/xwfs

Two classes of systems

Kraken

Keeneland

Darter

Beacon

Nautilus

HPSS


#30

The Kraken Supercomputer


Each compute node has:

• Two 2.6 GHz six-core AMD Opteron processors (Istanbul)

• 12 cores

• 16 GB of memory

• Connection via Cray SeaStar2+ router

• Cray Linux Environment (CLE) 3.1

• A peak performance of 1.17 PetaFLOP

• 112,896 compute cores

• 147 TB of compute memory

• A 3.3 PB raw parallel file system of disk storage for scratch space (2.4 PB available), with capacity of 30 GB/s.

• 9,408 compute nodes

• 3D torus interconnect.

Kraken Specs


Keeneland Supercomputer

#87Tuesday, October 15, 13

Nvidia Tesla GPU

GPU computing offers unprecedented application performance by offloading compute-intensive portions of the application to the GPU, while the remainder of the code still runs on the CPU.

GPU computing is the use of a GPU (graphics processing unit) together with a CPU to accelerate general-purpose scientific and engineering applications.

SKU # M2090

Form Factor PCIe, Gen 2, 8GB/s

Processor Cores 512

Peak Double Precision 665GFlops

Peak Single Precision 1330GFlops

Core clock speed 1.3 Ghz

Mem capacity 6GB

Mem BW 178 GB/s

Peak mem BW 320

Total cache 30MB

Board TDP <=225 Watts

Thermal cooling Passive



• Two 2.6 GHz eight-core Intel SandyBridge (Xeon E5) processors

• 16 cores

• 32 GB of memory

• 3 Nvidia M2090 (Tesla) GPU’s (665 GFlops) w/6GB device memory with ECC on, connected with PCIe-16 full bandwith

• HP SL250s Gen8 Cluster Platform

• Linux CentOS 6.2

• 11 compute racks

• A peak performance of 615 TeraFLOP

• 4,224 compute cores and 792 GPUs

• 8.4 TB of compute memory

• It uses the Medusa parallel file system

• 264 compute nodes

• Mellanox 384p, FDR Infiniband Switch

Keeneland Specs(KFS)


The Darter Supercomputer

#?Tuesday, October 15, 13



• 16 physical cores (32 w/hyper-threading)

• 32 GB of memory

• Cray Aries interconnect with 8GB/sec bandwidth

• Cray XC30 (Cascade)

• Cray Linux Environment 5.0 upo3

• 4 compute racks

• 23,936 compute cores w/hyper threading

• 24 TB of compute memory

• 334TB Sonexion parallel file system


• Cray Aries Interconnect

Darter Specs


The Beacon Supercomputer

#397#3 or?


Intel® Xeon Phi™ coprocessor 5110P (Knights Corner)

The Intel® Xeon Phi™ coprocessor (codenamed Knights Corner) is the first commercial product employing the Intel® Many Integrated Core (MIC) architecture. (The Intel® Xeon Phi™ coprocessor 5110P shown here employs passive cooling.)

SKU # 5110P

Form Factor PCIe card

Thermal solution Passively cooled

Peak Double Precision 1011GF

Max Number cores 60

Core clock speed 1.053 Ghz

Mem capacity 8GB

GDDR5 mem speed 5.0 GT/s

Peak mem BW 320

Total cache 30MB

Board TDP 225 Watts

Fabrication process 22 nm




• Four Intel Xeon Phi (MICs) 5110P coprocessors

• 16 physical cores (32 w/hyper-threading)

• 960GB local SSD storage

• 256GB of system memory and 8GB of GDDR5

• Cray CS300-AC Cluster (APPRO)

• Linux CentOS v6.2

• 4 compute racks

• 768 compute cores and 11,520 coprocessor cores

• 12 TB of system memory

• 73TB Total scratch SSD storage


• Infiniband FDR Interconnect

Beacon Specs


Nautilus Supercomputer

#?Tuesday, October 15, 13

(4x)SGI UV10 [Harpoon nodes]

• 32 Intel Nehalem 2.0GHz core

• 128 GB of memory

• 3 Nvidia Tesla GPUs

• SGI UltraViolet 1000

• Linux SLES 11.1

• 2 compute racks

• 1,024 2.0GHz Intel Nehalem cores

• 4 TB of shared memory

• Single System image

• 1.3 PB Medusa parallel file system

• Tons of visualization applications including Matlab

Nautilus Specs


HPSS archival System


• (3) Oracle/Sun/StorageTek Silos

• HPSS framework

• SL8500 tape libraries, each holding up to 10,000 cartridges, i.e. 10PB!

• The libraries house a total of (24) T10K-A tape drives for 500GB tapes, and, (36) T10K-B tape drives for 1 TB tapes

• Uses 1 TB cartridges (uncompressed)

• Each drive has a bandwidth of 120 MB/s.

HPSS Specs


overview of xsede and ut’s hpc resources · 2013. 10. 15. · • jülich supercomputing centre...

Documents