technion - israel institute of technology department of electrical engineering

Technion - Israel institute of technologydepartment of Electrical Engineering

High speed digital systems laboratory

Super Computer System

Midterm presentation

Developers : Anton Vainer and Atila FuadMentor : Mr Assad Haick

Dec 21, 2011

Super Computer System Agenda

Project Overview –Reminder Final system setup and configuration Memory map Startup flow and Driver operation Gantt Chart

Super Computer System Overview –Reminder

• For Validation of 10G Ethernet products we need a computer that can generate high speed traffic. Standard Computer don’t have the bandwidth needed.

• Super computers are very large, expensive and loosely coupled systems. In the project we will build a small tightly coupled super computer system with four computer chassis each with 1-4 CPUs.

• This super computer will be able to deliver the bandwidth needed to stress test 10G products.

LADi HaifaINTEL CONFIDENTIAL

10G Link Partners(Option – 1)

4 Agents run above the same TCP stack

Super Computer System Final system setup and

configuration

Slave Slave

SlaveMaster

PEX8617AIC RDK

PEX8617AIC RDK

PEX8617AIC RDK

PEX8617AIC RDK

PEX8617AIC RDK

PEX8617AIC RDK

PEX8648RDKPEX8648RDK

LADi HaifaINTEL CONFIDENTIAL

PLX 8617PCIe Gen II Switch160 GT/s16 Lane - 4 PortsFull line rate on all portsNon Blocking SW which can support both modes:• Transparent • Non-Transparent (NT)

Super Computer System Final system setup and

configuration

LADi HaifaINTEL CONFIDENTIAL

PLX 8648PCIe Gen II Switch480 GT/s48 Lane - 12 PortsFull line rate on all portsNon Blocking SW which can support both modes:• Transparent • Non-Transparent (NT)

Super Computer System Final system setup and

configuration

LADi HaifaINTEL CONFIDENTIAL8

Super Computer System Final system setup and

Super Computer System Final system setup and

configuration• Each one of the 3 slave computers has a 8617 16-lane PCIe switch configured as a non-transparent bridge.• Master computer is connected directly to 8648 48-lane PCIe switch with NT-Bridge.• Each slave NT-Bridge is connected to a similar 8617 card on the main board that is configured as clock isolation only (no NT-bridge) because of a bug in spread spectrum clock in Intel Chipsets.• Each slave NT bridge is configured via EEPROM to allocate

it’s own BAR0 and an additional 1MB for BAR2.

Super Computer System Memory map

NT-Virtual NT-Link

CPU 2-4 CPU 1BAR0 128k

BAR2 1M CPU # BAR2

Addresstranslate

allocated 16MFor TX data

1M BAR0Of the NIC

port 0

Switch

CPU 1 BAR2

Addresstranslate

Allocated 16M

Super Computer System Startup flow and Driver operation

* Slave machine: - Upon boot, the NT-Bridge (configured via EEPROM) will allocate 128k as BAR0 for itself and 1M BAR2 for the NIC. - Upon driver load, the driver will allocate 16M of memory for TX data.

* Master machine: - Upon boot, the NIC will allocate 1M BAR0 for itself. NT-bridge will allocate 128k BAR0 for itself and 16M BAR2 for TX data. - Upon driver load, the NT bridges will be configures via internal registers to mirror each slave machine BAR2 to master’s NIC BAR0 (to get NIC register access), and the allocated 16M to master’s NT-bridge BAR2 (to TX every slave machine's data).

Super Computer System Gantt Chart

Task Duratio

nStart Date

Finish Date

Studying the tools 30 1.11 30.11Design the setup and driver abilities 15 1.12 15.12Characterization presentation 5 11.11 16.11Setup bring-up 15 15.11 30.11Mid. Project Presentation 5 16.12 20.12Enabling of PCIe switch 20 20.12 10.1Implementing the driver 40 20.12 31.1Testing and validation 28 1.2 28.2Final project presentation 5 20.3 25.3