IBM RS/6000 SP POWER3 SMP

Jari JokinenPekka Laurila

IBM RS/6000 Series Series of advanced IBM Unix computers. Available from entry level to high-end

machines. Allows many different configurations. Current high-end configurations use

POWER3 type processors. Shared memory MIMD (Multiple

Instructions, Multiple Data) architecture.

POWER3 processor 64-bit PowerPC RISC-processor. Symmetric multi-processor. 64-bit addressability and double-

word integer operations. Eight execution units. High bandwidth memory with four

floating point operations per cycle.

POWER3 processor

POWER3 processor 200MHz processor achieves:

623 MFLOPS 1.1Gbps memory bandwidth

The technology allows processor speeds up to 500MHz.

Allows complex switch-based configurations to be built using POWER processor.

Node architecture SP POWER3 SMP high node

architecture. Up to 16 POWER3 processors per

node with up to 64 GB of memory. Scalable architecture allowing

system from one node up to 512 nodes to be built using same technology.

Node architecture Up to four processor cards. Each card has up to four processors. The Node Controller chips for data have

4GB/s bandwidth per processor and 16GB/s bandwidth to the Active Backplane Planar.

Memory and I/O functions have 16GB/s bandwidth to Planar.

Tree topology inside the node.

Node architecture

Connecting nodes SP Switch2 is used to connect nodes

into a supercomputer. Hardware + software = Communication

Subsystem (CSS)• Communication path, monitoring of the

switch hardware, controling the network, error detection and recovery action.

Multistage switching technology

SP Switch2 32 links, 16 for nodes and 16 for

other switches. For very big networks, switch boards

have to be connected together. Also available is an 8 node switch

board for those who do not need more than 8 nodes.

Switch board

Switch board

Switch board

Two switch boards connected

Topology 2 to 80 nodes can be connected

together with maximum of 5 switch boards using star topology.

Data passes through, at most, two switch boards.

Multiple boards connected

80 to 256 nodes When more than 80 nodes are

required, at least six switch boards must be used. Star topology does not allow this.

Instead, additional boards must be used as intermediate switch boards.

Intermediate switch boards

257 to 512 nodes If more than 257 nodes are required,

two frames of switch boards have to be used.

32 NSBs times 16 ISBs equals 512 nodes.

Operating System IMB’s own AIX operating system is

used in the machines based on this architecture.

64-bit Unix system. Each node has its own operating

system.

NERSC IBM SP RS/6000 Uses SP POWER3 SMP architecture. 208 16-processor nodes where 184

are actual computing nodes, 16 for fileserver (GPFS) with 20 Terabytes of disk space, 3 login nodes, 1 network node and 4 service nodes.

375MHz POWER3 processors.

NERSC IBM SP RS/6000 Currently fifth on the top500

supercomputer list. Usage restricted to scientific

projects funded by Department of Energy.

NERSC IBM SP RS/6000 Peak performance of 4.5TFLOPS Actual benchmarked performance:

With MPI, latency between nodes is 17 microseconds, within nodes it is 9 microseconds. Bandwidth between nodes is 60-400MB/s (16-2 nodes), within nodes 250-500MB/s (16-2 processors).

NERSC IBM SP RS/6000 MPI programming environment, C/C++,

FORTRAN Other compilers:

OpenMP, Java for AIX, Assembler for AIX Tools:

Parallel Operating Environment (POE) for AIX, LoadLeveler for job queueing, TotalView for debugging, HPMcount for monitoring hardware and Xprofiler for profiling serial and parallel applications.

Many mathematical libraries.