www.tttech.com copyright © 2009, tttech computertechnik ag. all rights reserved; may be published...

www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009

Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA

Wilfried Steiner

[email protected]

TTTech Computertechnik AG

MAPLD 2009 – Session A

CoMMiCS

www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009

Problem Statement

1. Reliable Synchronization

2. Multi-Hop Network Topology

3. TTEthernet - Realization in FPGA

4. Model-Based Development

TTE

1588

1588

Eth

TTE

TTE

TTE

Eth

TTE

TTE

TTE

TTE

TTE

TTE

TTE

Eth


Synchronization Strategy

Perfe

ct Cloc

k

Real Time

Com

pute

r T

ime

Slow Clock

Fast Clock

R.int

Mes

sage

Exc

hang

e

R.int

Mes

sage

Exc

hang

e

Clock Synchronization Service

Startup/Restart Service

Clock Synchronization Service is executed during normal operation mode to keep the local clocks synchronized to each other.

Startup/Restart Service is executed to reach an initial synchronization of the local clocks in the system.

Integration/Reintegration Service is used for components to join an already synchronized system.

Clique Detection Services are used to detect loss of synchronization and establishment of disjoint sets of synchronized components.


Two-Step Clock Synchronization Service

For now, let us assume that we operate in a single-hop network:

End Systems operate as Synchronization Masters/Clients.

Switches are configured as Compression Masters.

Synchronization Strategy operates in two steps for clock synchronization during normal operation mode.

Step1: Synchronization Masters send synchronization messages to Compression Masters.

Step2: Compression Masters send synchronization messages back to the Synchronization Masters and Clients.

SynchronizationMaster

Compression Master

ID 1 ID 2

ID 3

Step 1


SynchronizationClient



Compression Master

ID 5 ID 5

ID 5ID 5

Step 2


SynchronizationClient


ES 101dispatch

permanence

permanence

ES 102

ES 103

ES 104

Sw

itch

201

791 792 794

permanence

permanence

permanence380

380

380

380

380

380

380

380

dispatchpermanence

dispatchpermanence

dispatchpermanence

dispatch

dispatch

dispatch

dispatch

1110

1110

1110

1110

1000

0100

0010

0001

1000

0100

0010

1110

1110

1110

1110

scheduleddispatch_point_in_time round_trip_delay acceptance_window

dispatch_point_in_time

calculation of dispatch_point_in _time

collection oflocal clock readings

re-synchronization interval x x + 1x - 1


Four-Step Startup/Restart Service

Port 1send

receive

sendreceive

sendreceive

sendreceive

Port 2

Port 3

Port 4Fau

lty S

witc

h 20

1

Port 1send

receive

sendreceive

sendreceive

sendreceive

Port 2

Port 3

Port 4Cor

rect

Sw

itch

202

ES 101

sendreceive

Fau

lty E

S 1

01

sendreceive

ES 102

sendreceive

sendreceive

ES 103

sendreceive

sendreceive

ES 104

sendreceive

sendreceive

CS

CS

CS

CS

CA

CA

CA

CA

CA

CA

CA

CA

CA

CA

CA

CA

3101

3101

3101

3101

CSO

4103

4202

4202

4202

4202

4103

4103

4103

4202

4202

4202

4202Step1: Synchronization Masters send out Coldstart Frames (CS)

Step2: Compression Masters forward all CS frames to all ports

Step3: Each Synchronization Master that receives a CS frame will answer with a Coldstart Acknowledgement Frame (CA)

Step4a: Compression Masters will forward all CA frames to all ports (for multiple failure tolerant configurations).

Step4b: Compression Masters will forward only one compressed CA frame (for single failure tolerant configurations).


Formal Verification Activities (Synchronization Strategy)

TTEthernet Executable Formal Specification Using symbolic and bounded model checkers sal-smc and sal-bmc

Focus on Interoperation of Synchronization Services (Startup, Restart, Clique Detection, Clique Resolution, abstract Clock Synchronization)

Verification of Lower-Level Synchronization Functions Permanence Function

verified with the infinite-bounded model checker sal-inf-bmc

using disjunctive invariant and k-induction

Compression Function verified with the infinite-bounded model checker sal-inf-bmc

using abstraction and 1-induction

Formal Methods have been applied as early as in the requirements capturing phase

Finalization and Completion of the formal assessment within the CoMMiCS Project

Complexity Management for Mixed-Criticality Systems

European Communities FP7 project [FP7/2007-2013] no. 236701]

CoMMiCS


TTEthernet in a Nutshell

… is all about smart ways to integrate data flows with differentcharacteristics in a single communication infrastructure.

While various groups started with standard Ethernet and aimat enhancing it with real-time and fault tolerance properties,TTTech comes from the opposite direction:

Our know-how is in real-time and fault-tolerant data flows.

We are applying our know-how on top of Ethernet and optimize ourconcepts with respect to Ethernet features (switches, full-duplex, etc).

In principle we can enable any carrier protocol with time-triggered technology, …

… but there are some good reasons for using Ethernet.


The Motivation for Ethernet

Ethernet hardware is low cost.

Ethernet is a well-established open-world standard and very scaleable.

The OSI reference model gives a well-structured classification of concepts that can be built on top of Ethernet.

Existing tools can be leveraged as cost-efficient diagnosis tools.

As all messages in TTEthernet are standard Ethernet compliant, existing tools can be leveraged for time-triggered messages as well.

Standard web servers can be leveraged for maintenance and configuration.

TTEthernet allows hosting of IEEE 1588 clock synchronization clients.

Engineers learn about Ethernet at school.

Ethernet compatibility enables the usage of technology that is established, tested, and verified.


Different applications have different communication needs: fault, tolerance, real time, high speed, high bandwidth, low latency, low jitter, active redundancy, hot standby, …

TTEthernet provides following traffic classes for applications:

Time-Triggered (TT): Bandwidth in TTEthernet networks can be highly utilized due to the possibility of strictly deterministic (vs. probabilistic) traffic scheduling of high-priority periodic traffic.

Rate-constrained (RC): Event-triggered traffic with priority levels (ARINC 664) and transmission guarantees.

Non-critical standard Ethernet (BE): Low priority traffic (e.g. data download) served during network idle times; can also be scheduled explicitly.

Dataflow in TTEthernet

TTEthernet allows applications with different communication requirements to share a single physical network.


TTEthernet Integrated Dataflow

TT TTTT TT TTTT TT

3ms cycle

2ms cycle

3ms cycle 3ms cycle

2ms cycle 2ms cycle 2ms cycle

6ms Cluster Cycle

RC BE BE BE RC BE t

Sender

1 Switch/RouterReceiver

Sender

2

TT TT TT

3ms cycle 3ms cycle 3ms cycle

BE BE BE t

TT TT TT

2ms cycle 2ms cycle2ms cycle

BE BE RC BE

t

Dataflow – Integration - Time-Triggered (TT)

- Rate-Constrained (RC) - Standard Ethernet (BE)

Protocol Control Frames (PCFs) are transmitted on the same physical network as dataflow.


Integration Problem

When two (or more) messages compete for relay to the sameoutgoing port, the switch (or ES) has to serialize these messages.

If there are messages of same priority the messages will be serviced according First-Come First-Served.

What happens if there is a Data message (Data) in relay, when a Protocol Control Frame (PCF) becomes ready for relay?

If a Data message is relayed by a switch when a PCF arrives, the PCF message is delayed until the relay process of the Data message is finished.

Hence, in the worst case the PCF is delayed for a maximum sized Data message (plus other queued PCFs).

Protocol Control Frames record their delay through the network.

All components in the network that impose a delay on a PCF will add this delay into a field in the PCFs.

PCF

Data

PCF Data

Data PCF

PCF Data

Preemption:

Timely Block:

Shuffling:real-time

Contention:


Single-Hop Synchronization Flow

Compression MasterSynchronization

Master/Client

Frame Sendsm_send _pit


Frame Dispatchsm_dispatch _pit

Real Time

Local Clock

Frame Receivecm_receive _pit

Frame Scheduled Receivecm_scheduled_receive _pit

Frame Permanencecm_permanence _pit

Frame Compressedcm_compressed _pit

Frame Dispatchcm_dispatch _pit

Frame Sendcm_send _pit

Frame Receivesmc_receive _pit

Frame Permanencesmc_permanence _pit

Frame Scheduled Receivesmc_scheduled_receive _pit

(2 * max_transmission_delay + compression_master_delay)

(2 * max_transmission_delay + compression_master_delay) + clock_corrDelays

Transparent Clock

TC TC

Acceptance Window Acceptance Window

Compression Function

Message PermanenceFunction


1

2

3

4

5

6

7

8 9

11

10

The (Dispatch, Permanence) – Pair hides the network jitter for PCFs almost entirely.

~constantdynamic


Multi-Hop Synchronization Flow

Compression Master Synchronization Client

Frame Sendsmc_send _pit

Synchronization Client

Real Time

Local Clock

Frame Receivecm_receive _pit

Frame Scheduled Receivecm_scheduled_receive _pit

Frame Permanencecm_permanence _pit

Frame Compressedcm_compressed _pit

Frame Sendcm_send _pit


Frame Permanencesmc_permanence _pit

Frame Scheduled Receivesmc_scheduled_receive _pit

Transparent Clock

TC TC


Frame Sendsmc_send _pit

TC

From Synchronization

Master

To Synchronization

Master/Client

Frame Dispatchcm_dispatch _pit

Compression Function

Acceptance Window Acceptance Window



12

TC+

Multi-hop requires only a small add-on to the existing mechanism: an intermediate switch will not only consume the PCF, but also forward it.


Synchronization Master / Client i

FilterFrame

PermanenceSync StateMachine

Protocol Control Frames (PCFs) from the Compression Master flow in from the left.

PCFs produced by the Synchronization Master flow out on the right.

Filter: removes PCFs that are semantically incorrect (e.g. wrong type, outdated, etc.).

Frame Permanence: holds PCFs in buffers until their age equals the configured max. transparent clock value (respecting the transparent clock field as part of their age).

Sync State Machine: this module will process all the PCFs that made it past the Frame Permanence module as specified by the synchronization services. In particular this module is in charge of maintaining the local copy of the synchronized cluster time.


Synchronization Master / Client ii

The size of the Synchronization Unit depends upon others on the number of Synchronization Masters that have to be supported.

For sixteen Synchronization Masters:

the complete block (incl. interfaces to MACs) takes ~5,000 flip-flops (including diagnostics) and ~7,000 ALUTs

this is 12% and 15%, respectively, of the total size of a 3 channels GBits/s End System controller in an Altera Stratix III FPGA.

For four Synchronization Masters:

the block needs ~2,200 flip-flops and ~3,500 ALUTs.


TTEEvaluation System (2 channels, 1 Gbit/s)


TTEEvaluation System

Takes you from first evaluation steps to full-blown prototypes

4 Linux-based PCs with triple channel TTEthernet capability

Optical Gigabit Ethernet (1000Base-X) compliant interfaces

One or more high-performance TTEthernet switches with 4 ports and configuration, monitoring and diagnostic interfaces

Optical cables supplied for easy connection to existing LAN infrastructure or directly to the user’s development PC (needs 1000Base-X interface)

TTEthernet end system software libraries offering an API (application programming interface) to the TTEthernet protocol hardware

TTEthernet tools for network planning, schedule inspection, and configuration

PC-based monitoring tool supports logging/recording for analysis of network traffic and network-based applications

Power supply 110-240 VAC

Example application, documentation


TTEthernet Hardware Products - Summary

TTEDevelopment Switches TTEDevelopment Switch 1Gbit/s

TTEDevelopment Switch 100Mbit/s

TTEEnd Systems TTEEnd System PCIe

TTEEnd System PMC

TTEEnd System XMC

TTEEvaluation Systems TTEEvaluation System 1Gbit/s

TTEEvaluation System 100 Mbit/s

TTETest Equipment TTEMonitoring Switch 1Gbit/s

TTEMonitoring System



Model-Based Development


TTEthernet Executable Formal Specification i

SM1

SMk

SM2

CM1

CM2

CONNECTIONS

…

Current State

Ideal State

DIAGNOSIS

= or /=

FilterFrame

PermanenceCompression

MasterFrame


FilterFrame




Summary & Conclusion


Summary

Reliable Synchronization We showed the two-step clock synchronization service and the four-step

startup/restart service and discussed their failure-tolerance capabilities.

for Multi-Hop Networks We discussed the problem of integrating Control-Flow and Dataflow on the same

physical network.

We showed that the permanence function is a very simple mean to establish a strong interface with clean semantics for the synchronization services.

and its Realization in FPGA We introduced the conceptual design and discussed the implementation details in an

Altera Stratix III FPGA.

Model-Based Development We introduced the Executable Formal Specification and sketched how the worst-case

stabilization time of TTEthernet has been calculated.


QUESTIONS?

www.tttech.com copyright © 2009, tttech computertechnik ag. all rights reserved; may be published...

Documents

reliable synchronization

synchronization messages

loss of synchronization

initial synchronization

synchronization mastersclients

compression masters

fpga slide

commics slide