real-time ethernet and pci express do they fit together?

Real-Time Ethernet and PCI ExpressDo they fit together?

Michal Sojka, Rostislav Lisovy, Zdenek Hanzalek

Czech Technical University in Prague,Faculty of Electrical Engineering

Technicka 2, 121 35 Prague 6, Czech RepublicEmail: {sojkam1,lisovros,hanzalek}@fel.cvut.cz

RATE 2013 WorkshopDecember 3, Vancouver, Canada

M. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 1 / 14

Introduction

Real-Time Ethernet

I Real-Time = Deterministic

I TDMA – Deterministic medium access algorithm

I Also called “Time-triggered communication”

Slot 0 Slot 1 Slot 2

TDMA cycle

Slot 0

Node A Node B

Introduction

Real-Time Ethernet

I Real-Time = Deterministic

I TDMA – Deterministic medium access algorithm

I Also called “Time-triggered communication”

Slot 0 Slot 1 Slot 2

TDMA cycle

Slot 0

Node A Node B

Introduction

Time Synchronization

I TDMA requires synchronized time between nodes

Slot 0 Slot 2

Slot 1

Node A

Node B

I Synchronization accuracy influences protocol efficiency

Introduction

Time Synchronization

I TDMA requires synchronized time between nodes

Slot 0 Slot 2

Slot 1

Node A

Node B

I Synchronization accuracy influences protocol efficiency

Introduction

Precision Time Protocol (PTP)

I Standardized as IEEE 1588

I Can take advantage of hardware timestamping of transmittedor received frames

I Hardware timestamping is available on many modern NICs1

I Synchronization accuracy: < 1µs

I Sounds good!

1Network Interface CardM. Sojka, et al. Real-Time Ethernet and PCI Express: Do they fit together? RATE 2013 4 / 14

Introduction

PC Hardware

I Cheap, powerful, extendable ⇒ attractive

I Popular in academic papers about real-timeEthernet

I IEEE 1588 NIC + Real-Time OS = Perfectsolution for networked real-time systems

Is that true?

Introduction

PC Hardware

I Cheap, powerful, extendable ⇒ attractive

I Popular in academic papers about real-timeEthernet

I IEEE 1588 NIC + Real-Time OS = Perfectsolution for networked real-time systems

Is that true?

Modern PC

Modern PC Architecture = PCI Express Architecture

Core 0

Root Complex RAM DDR3

PCIe GraphicsAdapter Slot

PCIex16

DMI / PCIePCH (Chipset)

Core 1 Core 2 Core 3

Serial / Parallelport; Keyboard,Mouse PS/2

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

PCI Express (PCIe)

I Communication betweenCPU, memory andperipherals

I Packet-based serialnetwork with switches

I Shared resource

I Differences between PCIeslots (bandwidth, latency,jitter)

Modern PC

Modern PC Architecture = PCI Express Architecture

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

PCI Express (PCIe)

I Communication betweenCPU, memory andperipherals

I Packet-based serialnetwork with switches

I Shared resource

I Differences between PCIeslots (bandwidth, latency,jitter)

PCI Express Latencies

Experiments with PCI Express Latencies

I Intel 82576 NIC + Ivy Bridge CPU + bare-metal microhypervisorI Unloaded and loaded system

Reading a 32 bit register from the NIC

I Worst-case latency: 3.5µsI Jitter: 3µs

I NIC clock register: 64 bit (⇒ ×2)

Timing comparison

I CPU reads the NIC clock: worst-case latency 7µs, jitter 6µsI NIC clock has accuracy of < 1µs

I Compare: Parallel PCI, 10 years ago ⇒ average latency and jitterbelow 1µs.

Timing comparison

Sources of PCI Express Latencies

PCIe topology – Slots connected directly to the CPU have lowerlatency than slots provided by the “chipset”

Active State Power Management (ASPM) – Technology thatsaves power by switching off idle PCIe links

Unrelated traffic – Load initiated by other CPUs or PCIe devices inthe system

Sending an Ethernet Frame

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

Core 0

PCIex16

PCIeSwitch

PCIe to PCIBridge

PCIeSwitch

PCISlots

Frame Transmission Latency

I 166 bytes long frame: worst-case latency 20µs, jitter 10µs

I 1200 bytes long frame: worst-case latency 22µs, jitter 10µs

I ≈ 3 PCIe transactions per frame

Time Comparison

PCIe-caused TX jitter

Frame TX time @ 100Mbps

Frame TX time @ 1Gbps

0 5 10 15 20 25

166 bytes

1200 bytes

Time [µs]

Packet size

Worst-case TX latency GFX slotIO slot

Conclusions

Consequences

1. Software implementation of time-triggered protocols cannotachieve network utilization above about 30% (@ 100 Mbps).

2. Precise measurement applications running on a CPU cannotuse the clock precision offered by PTP. Even unloaded systemexhibits jitter of about 5µs in certain setups.

3. Some applications might benefit from having synchronizedexecution of CPU tasks with network time. This is possiblebut one needs to account for high PCIe jitters.

Conclusions

Consequences

Conclusions

Consequences

Conclusions

What Can Be Done About It?

I Benchmark PCIe latencies and jitter of different slotsI We work on a Linux tool that can do that without special

experimental setup

I PCIe specification: Virtual Channels

I Different QoS parameters for each channelI Isochronous traffic – bandwidth reservationI Not available in common PC hardware (?)

I PTP over PCI Express?

Conclusions

Thank you!

R. Lisovy, M. Sojka, and Z. Hanzalek, “PCI Express as a killer ofsoftware-based real-time Ethernet,” in Real-Time Networks, The

12th International Workshop on, July 2013.

real-time ethernet and pci express do they fit together?

Documents