Who We Are?Who We Are?

Detector Building Group of Detector Building Group of KFKI-RMKIKFKI-RMKI

(Research Institute for Particle and Nuclear Physics), (Research Institute for Particle and Nuclear Physics), Budapest, HUNGARYBudapest, HUNGARY

Our ResultsOur ResultsPrevious projects: Previous projects: Designing Fibre Channel test equipments (1994-1998)Designing Fibre Channel test equipments (1994-1998)

Portable, 266 Mb/s Fibre Channel Tester Portable, 266 Mb/s Fibre Channel Tester Fibre Channel Preprocessor for Logic AnalyzersFibre Channel Preprocessor for Logic Analyzers

Designing high-speed data transmission interfaces Designing high-speed data transmission interfaces for CERN detectors (1996 - ...)for CERN detectors (1996 - ...) Protocol design and verificationProtocol design and verification FC, GbE, physical layer components FC, GbE, physical layer components Hardware designHardware design Software: linux drivers, test programs, program librarySoftware: linux drivers, test programs, program library

S-LINK (CERN)S-LINK (CERN)

S-LINK interface cards for CERN (mostly used at ATLAS)S-LINK interface cards for CERN (mostly used at ATLAS) simple, unidirectional link interface simple, unidirectional link interface first successful version was designed by KFKI-RMKIfirst successful version was designed by KFKI-RMKI newer versions, now: 2.5 Gb/snewer versions, now: 2.5 Gb/s

Also used at MPI, Garching and at several HEP and other Also used at MPI, Garching and at several HEP and other scientific institutes all over the worldscientific institutes all over the world

ALICE Detector Data Link (DDL)ALICE Detector Data Link (DDL)

DDLDDL

Detector Data LinkDetector Data Link (DDL) (DDL) for the ALICE detector at CERNfor the ALICE detector at CERN

2.5 Gb/s, duplex link2.5 Gb/s, duplex link advanced featues advanced featues

Test devices for DDLTest devices for DDL (Front-end emulator, DDL link emulator, etc.)(Front-end emulator, DDL link emulator, etc.)

PCI Read-Out Receiver Card PCI Read-Out Receiver Card (RORC)(RORC) for DDL for DDL 1st version: 33 MHz, 32-bit PCI card 1st version: 33 MHz, 32-bit PCI card 2nd version: 66 MHz, 64-bit PCI 2nd version: 66 MHz, 64-bit PCI

with 2 integrated DDL interfaceswith 2 integrated DDL interfaces

Front-endElectronics

Front-endElectronics

DDLSource

InterfaceUnit

)SIU(

DDLSource

InterfaceUnit

)SIU(

TAP

DDLDestination

InterfaceUnit(DIU)

DDLDestination

InterfaceUnit(DIU)

Read-outReceiver

Card(RORC)

66 MHz64-bit

PCI

Read-outReceiver

Card(RORC)

66 MHz64-bit

PCI

Front-end Bus32 bit

JTAG BST lines

32 bit

32 bit

4

1

FEE - SIU InterfaceDIU -RORC Interface

optical cable, max 350 m

DDL

DDL InterfacesDDL Interfaces

DDL FeaturesDDL FeaturesInterface:Interface: Full duplex 32-bit data path on the destination interface (DIU card) Full duplex 32-bit data path on the destination interface (DIU card) Half duplex 32-bit data path on the source interface (SIU card) Half duplex 32-bit data path on the source interface (SIU card) Full duplex flow control (XON/XOFF) Full duplex flow control (XON/XOFF) Interface clock up to 66 MHz (easy integration with PCI 66) Interface clock up to 66 MHz (easy integration with PCI 66) 264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.)264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.)

Implementation:Implementation: Duplex LC optical link up to 300 mDuplex LC optical link up to 300 m 2x FC or 2x GbE physical layer components2x FC or 2x GbE physical layer components Small Form Factor Pluggable (SFP) optical transceivers Small Form Factor Pluggable (SFP) optical transceivers Bit error rate < 10 Bit error rate < 10 -12-12 Robust error detection: very low undetected bit error rate < 10Robust error detection: very low undetected bit error rate < 10-40-40 Automatic link synchronization and management Automatic link synchronization and management

Extras:Extras: Stand-by support (low power consumption) Stand-by support (low power consumption) In-system reconfiguration /  Remote system upgrade In-system reconfiguration /  Remote system upgrade Monitoring of the aging of laser diode of optical transceiversMonitoring of the aging of laser diode of optical transceivers JTAG Boundary Scan Test interface for the Front-End electronics JTAG Boundary Scan Test interface for the Front-End electronics

Outlook to PCI ExpressOutlook to PCI Express

PCI Express (formerly 3GIO)PCI Express (formerly 3GIO)a „third generation” high performance I/O busa „third generation” high performance I/O bus

(1st generation: ISA, EISA, VESA, 2nd generation: PCI, PCI-X)(1st generation: ISA, EISA, VESA, 2nd generation: PCI, PCI-X)

PCI evolution (PCI, PCI-X, PCI-Express)PCI evolution (PCI, PCI-X, PCI-Express)PCI Express is software compatible to PCI and PCI-XPCI Express is software compatible to PCI and PCI-X

PCI EvolutionPCI Evolution PCI PCI „„multi-drop” parallel busmulti-drop” parallel bus conventional PCI: 33 MHz, 32-bit, max 4-5 card slots per busconventional PCI: 33 MHz, 32-bit, max 4-5 card slots per bus newer versions: 66 MHz, 64-bit, 1 (max 2) card slots per bus (!)newer versions: 66 MHz, 64-bit, 1 (max 2) card slots per bus (!)

PCI-X PCI-X parallel bus, backward compatible (hwparallel bus, backward compatible (hw and sw and sw)) 66 MHz , 32/64-bit, 66 MHz , 32/64-bit, max 4-5 card slots per busmax 4-5 card slots per bus 133 MHz, 32/64-bit,133 MHz, 32/64-bit, max 1-2 card slots per busmax 1-2 card slots per bus 266, 532 MHz versions: max 1 card slot per bus (!)266, 532 MHz versions: max 1 card slot per bus (!)

Buses can be bridged to each other (complex, expensive)Buses can be bridged to each other (complex, expensive)

PCI-XPPCI-XP It is still a local /IO bus, a „PCI bus”, but the connections It is still a local /IO bus, a „PCI bus”, but the connections

between devices are serial, point-to-point interconnectionsbetween devices are serial, point-to-point interconnections devices are interconnected via switch(es)devices are interconnected via switch(es) large number of devices can be interconnectedlarge number of devices can be interconnected highly scalable, hot-plug, hot-swap, QoS, etc.highly scalable, hot-plug, hot-swap, QoS, etc.

Limitations of PCI and PCI-XLimitations of PCI and PCI-X

Conventional 33 MHz PCIConventional 33 MHz PCI system system lowlow bandwith to nowada bandwith to nowadayy’’ss needs needs

66 MHz, 133 MHz (PCI-X), ...66 MHz, 133 MHz (PCI-X), ... onlonly fewy few devices can be interconnected devices can be interconnected (onl(only 1 or two) on a single bus y 1 or two) on a single bus

because of the because of the strict strict electrical load and timing constraintselectrical load and timing constraints

Further limitations of PCI architectureFurther limitations of PCI architecture inefficient solutions in:inefficient solutions in:

data transfer cycles (wait states)data transfer cycles (wait states) accessing of system memoryaccessing of system memory interrupt handlinginterrupt handling error handling error handling

PCI ExpressPCI Express

However, the basic problems of a parallel bus systemHowever, the basic problems of a parallel bus system(electrical load and timing constraints, lack of hot-(electrical load and timing constraints, lack of hot-pluggability, lack of scalability, etc.) can be solved pluggability, lack of scalability, etc.) can be solved only by a complete redesign of the architecture. only by a complete redesign of the architecture.

This resulted in PCI Express. This resulted in PCI Express.

Serial point-to-point interconnect between two devicesSerial point-to-point interconnect between two devices

1x, 2x, 4x, 8x, 12x, 16x, 32x type links1x, 2x, 4x, 8x, 12x, 16x, 32x type links 1, 2, 4, ... 32 bidirectional signal pairs (lanes)1, 2, 4, ... 32 bidirectional signal pairs (lanes) 1 lane: 2.5 Gb/s now, up to 10 Gb/s later1 lane: 2.5 Gb/s now, up to 10 Gb/s later

Low voltage, differential signaling (LVDS)Low voltage, differential signaling (LVDS) AC coupledAC coupled Data is encoded: 8B/10BData is encoded: 8B/10B

The PCI Express LinkThe PCI Express Link

Device A Device B(e.g. a switch)

a PCI Express link

1 to 32 lanes

BandwithBandwith

Scalable: mScalable: more lanes ore lanes /link: higher bandwith /link: higher bandwith 2.5 Gb/s per lane, 8B/10B encoding2.5 Gb/s per lane, 8B/10B encoding Simultaneous traffic in both directionsSimultaneous traffic in both directions

1x type link1x type link 500 MByte/s aggregate bandwith500 MByte/s aggregate bandwith 250 MByte/s per direction250 MByte/s per direction

32x type link:32x type link: 16 GByte/s aggregate bandwith16 GByte/s aggregate bandwith (8 GByte/s per direction) (8 GByte/s per direction)

Backward compatibilityBackward compatibility

Supports familiar PCI transactions such as memory Supports familiar PCI transactions such as memory read/write, IO read/write and configuration read/write read/write, IO read/write and configuration read/write

Same memory, IO and configuration address space Same memory, IO and configuration address space as in PCI and PCI-Xas in PCI and PCI-X

Existing OSs and driver software will run in a Existing OSs and driver software will run in a PCI Express system without any modifications PCI Express system without any modifications

Supports chip-to-chip interconnect and board-to-board Supports chip-to-chip interconnect and board-to-board interconnect via cards and connectors similar to the present interconnect via cards and connectors similar to the present PCI systemsPCI systems

PCI Express motherboard will have a similar form factor to PCI Express motherboard will have a similar form factor to existing ATX motherboards in PCsexisting ATX motherboards in PCs

Improvements, BenefitsImprovements, Benefits Fast, highly scalable, serial point-to-point interconnect between two devices Fast, highly scalable, serial point-to-point interconnect between two devices

bytes striped accross the lanes bytes striped accross the lanes more lanes per link: faster transmission more lanes per link: faster transmission

Packet-based communication protocol Packet-based communication protocol Packets are transmitted seriallyPackets are transmitted serially CRC embedded in each packet (auto retry: CRC embedded in each packet (auto retry: link levellink level error error correctioncorrection)) Buffer-to buffer (link level) flow control Buffer-to buffer (link level) flow control no need the packet retry no need the packet retry

Message Signaled Interrupt (MSI) architectureMessage Signaled Interrupt (MSI) architecture

No side-band signalsNo side-band signals hot-plug, error handling, interrupt signaling and else are accopmlished in-bandhot-plug, error handling, interrupt signaling and else are accopmlished in-band

Multiple devices are interconnected via switches Multiple devices are interconnected via switches Large number of devices can be connected together in a systemLarge number of devices can be connected together in a system

Much fewer pins per device package Much fewer pins per device package Reduces chip and board design cost and design complexity Reduces chip and board design cost and design complexity

Reconfigurable, hot-plug, hot-swap, improved power management, etc.Reconfigurable, hot-plug, hot-swap, improved power management, etc.

Quality of Service (QoS) features: Traffic Classes, Virtual ChannelsQuality of Service (QoS) features: Traffic Classes, Virtual Channels

Configuration address space of devices is extended from 256B to 4KB Configuration address space of devices is extended from 256B to 4KB This needs new softwareThis needs new software

Outside the box?Outside the box?

Initial focus of usage: Initial focus of usage: inside the box inside the box

Add-in card

switch

PCI-XP links

It is expected that later It is expected that later iit will t will also be used also be used ooutside the box utside the box for I/O expansionfor I/O expansion

Add-in card

switch

PCI-XP links

connector

opticalcable

PCI-XP link

Mechanical Form FactorsMechanical Form Factors

Connector and daughter card form factors are under specificationConnector and daughter card form factors are under specification

The main type of connectors is very similar to the present PCI card edge connectorThe main type of connectors is very similar to the present PCI card edge connector

A 1x type card can be inserted in a x4 type slot, a.s.o.A 1x type card can be inserted in a x4 type slot, a.s.o.

Like with PCI, there will be other form factorsLike with PCI, there will be other form factors Server I/O moduleServer I/O module Mini PCI Express card and connector (e.g. for notebooks)Mini PCI Express card and connector (e.g. for notebooks) mezzanine type cardmezzanine type card NEWCARDNEWCARD (will replace CardBus PC card), etc. (will replace CardBus PC card), etc.

Compete or complete? Compete or complete? PCI Express will coexist with PCI / PCI-X in the same systemPCI Express will coexist with PCI / PCI-X in the same system

PCI ExpressPCI Express intends to replace AGP (graphics card if), intends to replace AGP (graphics card if), but will not replace Serial ATA (hard disk, CD, etc)but will not replace Serial ATA (hard disk, CD, etc)

HyperTransportHyperTransport onboard chip-to-chip interconnectonboard chip-to-chip interconnect said to be complementary to said to be complementary to PCI ExpressPCI Express

RapidIORapidIO as an onboard chip-to-chip interconnect: complementary to PCI-Expressas an onboard chip-to-chip interconnect: complementary to PCI-Express as a system interconnect: a competitive technologyas a system interconnect: a competitive technology

1394b, USB2.0, Fibre Channel, Gb Ethernet, Infinband, etc.1394b, USB2.0, Fibre Channel, Gb Ethernet, Infinband, etc. PCI ExpressPCI Express cancan be a system interconnect that bridges these technologies be a system interconnect that bridges these technologies Will it be?Will it be?