spacecraft architectures based on...

www.tttech.com

Ensuring Reliable Networks

Copyright © TTTech Computertechnik AG. All rights reserved.

SPACECRAFT ARCHITECTURES BASED ON

DETERMINISTIC ETHERNET

Pasadena, 17th Dec 2014

Mirko Jakovljevic, Christian Fidi

[email protected]

[email protected]

December 17th, 2014

mailto:[email protected]



www.tttech.com



About TTTech

ISO 26262

Automotive

IEC 61508

Industrial

EN 13849

Off-Highway

DO 254/178

Aerospace

IEC 60601

IEC 62304

Medical

Market specific safety certification for integrated embedded systems

Boeing 787

NASA Orion

Audi A8

Airbus A380

Bombardier

CSeries

Embraer Legacy

450 / 500

Distributed Embedded Platforms and

Deterministic Networks from TTTech

www.tttech.com



TTTech Focus Integrated Embedded Platform

Integrated Platform

Ethernet Backbone

RTOS

Middleware

App

RTOS

Middleware

App

RTOS

Middleware

App

RTOS

Middleware

App

(RT)OS

Middleware

App

SW

Pla

tform

/

Ab

stra

ctio

n /

FT

Layers

/

Pla

tform

Serv

ices

Syste

m

Inte

gra

tion

Inte

rfacin

g

Application-

Specific

Functions

www.tttech.com



Deterministic Networks

Requirements for deterministic networks

• More bandwidth alone does not solve

QoS challenges

• Known (maximum or fixed) end-to-end

latency

• Bounded and small jitter

• Proper peak-load handling

• Proper handling of delays and faults in

communication

• Objective: Manageable design of

integrated embedded systems and

critical functions

www.tttech.com



Interfacing in Advanced Integrated Systems:

Different Traffic Classes /

Different Functions on Shared Resources

Periodic Data Streams

Integrated System Platform

(Deterministic Ethernet /

TTEthernet)

Periodic RT Control Loops

hard RT & RT

Periodic Media

Streams

Alarm/Protection RT Functions or Critical Event

Messages

(Traffic Bursts)

Soft-Time Functions

AperiodicTraffic and Traffic Bursts

* Typically not integrated with critical functions

www.tttech.com



Resource Sharing Challenges: RT

Computing in Integrated Systems

• Resource sharing among different integrated functions shall be

periodic and non-blocking

– Failure of one function shall not influence other functions or

generate resource starvation and deadlocks

– Resource sharing shall be carefully planned at design time, but in

„embedded clouds“ unknown „unknowns“ may exists

IMA (Integrated Modular Avionics or Arch.): • closed system, safety-/time-critical, real-time, deterministic

• Need to know about all functions and their resource

requirements: critical and non-critical, to setup the system

Distributed IMA / „Embedded Cloud“ Computing:

• (ideally) open system, generic architecture

• time- and/or safety-critical, hard RT, deterministic for

critical function

• Need to know only about critical functions in the system, to

setup the system

Scalable RT / HPC Computing

Critical IoT Infrastructure / Advanced C4 Systems

Integrated Modular Architectures with Hard RT

Reconfigurable Open Generic Architecture

Mixed Criticality Systems

www.tttech.com



Advanced System Integration

and Network Capabilities

Network capabilities

Model Of Computation and Communication

Distributed SW Platform Design

Application SW (Function) Design Methodology

System Lifecycle Costs

Architecture Design around the limitations of supported MoCC and network capability

System Lifecycle Costs

…Determine complexity!!!

www.tttech.com



Ethernet and

Deterministic Ethernet

(TTEthernet)

www.tttech.com



Space Programs Using Ethernet

www.tttech.com



A family of frame-based standards for LAN/MAN networks by

IEEE 802

• Standard physical medium

• Set of medium access

control rules with fair arbitration

• Variable size packets

in Ethernet format

• IEEE 802.1 focuses on Layer 2 QoS enhancements (traffic classes)!

802.3 focuses on bandwidth growth!

• Ethernet capabilities change over time! Not a monolithic standard!

• Ethernet device datasheets provide the list of supported functions and standards!

What is Ethernet, really?

Statistical Multiplexing

(Asynchronous Communication)

Best Effort

Traffic

Priority-driven VLAN Traffic

(802.1Q)

www.tttech.com



SAE & ARINC Standards for

Critical Ethernet Networking

SAE International (over 36000 standards and 138.000 members)

Networking standards: ARINC429, ARINC629, ARINC659, ARINC664(AFDX), SAE AS15531 (MIL-1553), ARINC825 (CAN), SAE J1939 (CAN), SAE AS5643 (Firewire), SAE AS6003 (TTP), SAE AS6802 (TTEthernet), SAE AS 4075A (HSRB), SAE AS5659 (WDM LAN), SAE AS5653A (MIL-1760) …

Typically SAE provides original networking standards, or network services / profiling to 3rd party (e.g. IEEE, …) networking standards to enable their application in critical infrastructure and integrated system applications.

SAE ITC Aviation Industry Actvities

(ARINC Standards) Focus Commercial Aviation / Integrated Systems and

Architectures / Datalinks

Driven By: Airliners / Aerospace Industry

SAE Standards Focus Aerospace/Space/Defense/Automotive/ Commercial vehicle / Integrated Systems and

Architectures

Driven By: Aerospace / Automotive / Transportation Industry

ARINC664 SAE AS6802

www.tttech.com



An Open Standard for Space

(Released 2011)

(In work since 2012)

www.tttech.com



Time-triggered extensions for standard

switched Gigabit-Ethernet

• Startup

• Recovery

• Robust fault-tolerant

distributed clock

• Foundation for design

of scheduled /synchronous

traffic class)

Extensions for Time-, Safety-, Mission Critical

Applications & IEEE Ethernet

Makes Ethernet viable for safety-critical distributed applications!

www.tttech.com



Traffic Class for Synchronous Communication

with defined QoS

System time available on switches and end stations

• Scheduled traffic can have fixed latency and µs-jitter

• Switch knows when the message is forwarded

By controlling jitter we also minimize

latency for critical streams

A large portion of latency in time-

sensitive rate-constrained

communication is the jitter!

www.tttech.com



Ethernet: Virtual links for Robust

Bandwidth Partitioning

• ARINC664 and SAE AS6802 QoS Layer 2 rely on virtual links (VLs) with

defined QoS and timing performance

• VLs emulate point-to-point connections in integrated architectures

• ARINC: max. latency per VL, SAE: fixed latency per VL

Note:

• Design of critical integrated systems not viable without VLs (VLANs cannot do the job!)

• Synchonous VL (prereserved bandwidth not used if no message sent)

• Asynchronous VL (require permanent reservation)

E/S

E/S E/S

...... ......

E/S

VL1

VL1VL4

E/S

E/S

VL22

VL21

V3

E/S

E/S E/S

E/S

...... ......

EthernetNetworkEthernetNetwork

E/S E/S

VL1

VL1 VL1

VL21

V3

VL4

VL22

www.tttech.com



Distributed Fault-tolerant

Synchronization

Robust algorithm based on exchange of asynchronous IEEE 802.3 messages

Synchronizes local clocks – system time (!)

• no wall clock (external time source - e.g. GPS) required

Fail-operational:

• tolerates multiple faults

• tolerates byzantine synchronization faults

• no search for best master (distributed clock!)

• Provides defined worst-case synchronous startup & recovery time (in ms)

www.tttech.com



Two-Step Fault-Tolerant Synchronization

Protocol Control Frames called “Integration Frames” are used to perform all

synchronization functions. They are transmitted accordingly:

Comp Sync

Sync

Sync

Comp

Comp

The Synchronization “Masters” send Integration Frames at the beginning of each Integration Cycle. The timing of these frames is used for the “voting”

The Compression “Masters” send Integration Frames to everybody, timing them in a special way so that everybody can correct their clocks.

Comp Sync

Sync Comp

Comp Sync

www.tttech.com


Copyright © TTTech Computertechnik AG. All rights reserved. Page 18

TTEthernet Traffic Partitioning

www.tttech.com



Ethernet Standards Incorporated in TTTech Network

Devices (Aerospace)

IEEE

802.1D

Layer 2 Switching

IEEE

802.1Q

VLAN Aware Bridge

Packet Priority (QoS)

ARINC664

Part 7

ARINC664 Virtual Links

(Asynchronous VL)

Policing

SAE

AS6802

Time- Triggered Virtual Links

(Synchronous VL)

Fault-Tolerant Clock

Synchronization

AFDX (ARINC664) and TTEthernet (SAE AS6802) Network Devices – Switches + Endsystems

Best Effort Asynchronous Synchronous

www.tttech.com



Distributed Platforms &

System Integration Capability

• (A) Platform Abstraction Middleware and Services

• separating application from architecture, simplifying distributed application design

• (B) Deterministic Network

• real-time communication guarantees, bandwidth partitioning and congestion management

• defined interaction, interfacing and separation among different distributed functions

• inter-partition communications (IPC) among different modules OR shared memory emulation

Distributed System with Sensors, Actuators (Effectors)

and Hard Real-Time Control Loops

App1a App1b App2 AppN

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

Distributed Embedded Computing Platform

Deterministic Network (B)

IPC & Platform Abstraction Middleware (A)

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

T

a

s

k

System-level IPC

AND/OR

Shared System Memory

...

Partitions

Module 1

Partitions

Module N

www.tttech.com



Key Design Ingredients: Virtualization in

Advanced Integrated Systems

MoCC:

Model of Computation/

Communication

(TTA and L-TTA)

Computing:

Time/Space-Partitioning

Network:

Robust Bandwidth

Partitioning And Virtual Links

(VLs)

www.tttech.com



MoCC (Models of Computation and Communication) support the integration of critical functions on shared embedded resources

L-TTA (asynchronous model, but works on local time)

• Cyclic/periodic processing of data based on local

clock/timer

• For RT control loops, defined max latency required

(asynchronous VL!)

• Enable deterministic control loop performance viable –

limits on hard RT performance

• Application Domains: Aerospace IMA/Railway

Signalling/Nuclear • Note: GALS is L-TTA with several partitionins per LRM

TTA (synchronous model, works on system time)

• Cyclic/periodic processing of data based on system time

• For simple integration of hard RT control loops, fixed

latency (synchronous VL!)

• Full hard RT performance

• Application Domains:

Aerospace/Space/Automotive/Railway-Rolling Stock

www.tttech.com



Virtual links for Robust Bandwidth Partitioning:

Impact on System Architectures

Asynchronous VL

Supports

L-TTA

Real-Time Performance

Closed Systems

(predefined critical and non-critical function performance)

Synchronous VL

Supports TTA and L-TTA

Enables „Embedded Cloud“

System-Wide

Hard RT Performance due to fixed latency

(SW function separated from controlled object)

Open Systems

(predefined critical function performance, arbitrary non-critical performance)

www.tttech.com



Deterministic Ethernet with Robust

TDMA Partitioning

DISTRIBUTED

FUNCTION 2

DISTRIBUTED

FUNCTION 4

DISTRIBUTED

FUNCTION 1

DISTRIBUTED

FUNCTION 3

IEEE802.3 Ethernet network

(Office LAN)

TTE

Partitioned OS

(e.g. VxWorks ARINC653)Partitioned OS

(e.g. VxWorks ARINC653)

Partitioned OS


Linux

Server

Windows

PC

Windows

PC

F1

F1 F1

TTE

TTE

Partitioned OS


Partitioned OS


Partitioned OS


Partitioned OS

(e.g. VxWorks ARINC653)F2 F2

F2F2

TTE

Partitioned OS


Partitioned OS


F3

F3

TTE

Partitioned OS


Partitioned OS


Partitioned OS


Partitioned OS


F4F4F4

F4

Equivalent to physically separated Ethernet subnetworks Embedded system virtualization (time-critical/time-sensitive/soft-time)

Allows "slicing" of shared computing/networking resources

Design of safety-/time-critical functions in

a distributed integrated systems

HINT:

VLANs do not support the virtualization of time-critical

functions! TDMA communication capability is required!

www.tttech.com



Industry Trend:

Time/Space Partitioning

• Multiple SW APPs are executed in

• Time and Space Partitions on

• A high performance, low cost HW

(SoCs)

OS1

APP

Linux OS3

APP APP

Hypervisor / TSP OS

SoC (CPU, FPGA, MEM, …)

www.tttech.com



Industry Trend:

System Level Partitioning

Win

APP

Linux OS

APP APP

Mem Mem Mem

Hypervisor / ARINC 653

Strong Partitioning

• Bandwidth partitioning at the network level

• Bandwidth partitioning supported at the switch and E/S level

• Memory partitioning at the E/S Level

• Bandwidth to memory mapping at the E/S based on virtual links

Bandwith Partitioning at

Switch Level

• Bandwith to Memory

Partitioning mapping at

E/S based on VLs

• Redundancy

management

www.tttech.com



Embedded Virtualization Cookbook for

Ethernet-based Integrated Systems

• Take care of resource use in

critical functions

• Use IMA methodology

• L-TTA/GALS/TTA, Asynch. VL

and/or Synch. VL, Partitioned OS

• Ensure integration of hard RT, RT and

soft-time functions

• Mixed L-TTA, TTA model

• Synchronous VL enables

• Full control of jitter and latency

• Network devices know exactly

what is going to happen with

hard RT and RT traffic

• Network devices know exactly

when the resources are free for

soft-time

• Non-critical/soft-time functions

can take care of themselves

• They use remaining resources

as background tasks

Embedded System Virtualization

MoCC (Model Of Computation And Communication)

L-TTA / TTA Priority/Event-driven

Non-Critical/Soft-Time Functions

Time Partitioning of Computing Resources

Network Bandwidh Partitioning with defined QoS (Virtual Links)

Priority-Driven VM and Task Execution

Statistical Network Bandwidh Multiplexing (VLAN, best effort)

Synchronous/ Hard RT

Asynchronous/ RT

Asynchronous

AsynchronousVirtual Links (VL)

SynchronousVirtual Links (VL)

Critical Functions (Time-Critical)

Technology Baseline

www.tttech.com



TTEthernet: Benefits

• Designed for N-redundant FT systems (typ. double or triple redundant

networks)

• Reduce the complexity of functional interactions

• Built-in mechanisms for FT synchronization

• Autonomous and scheduled operation

• Critical function timing defined @ network layer

• Influences design methodology and resource scheduling

• Simplifies software design, layering and partitioning

• Supports full separation and layering of temporal and functional behavior

in the system

• Enables design of „flat“ Distributed IMA Architectures (generic and

scalable architectures)

www.tttech.com



Supporting Slides

www.tttech.com



Complexity in Integrated Systems:

Synchronous vs. Asynchronous

Active standby avionics system model with three components…

• Synchronous model: 185 reachable states (~2x102)

• Asynchronous model & communication with no latency: >3x106 states

• Asynchronous model with varying communication

latency: The number of

reachable states could not

be calculated with 8Gb RAM…

https://www.ideals.illinois.edu/bitstream/handle/2142/17089/pals-formalization.pdf?sequence=2

>108-1010

???

The number of system states in an

integrated systems can be very

high…

And this is still a relatively simple

system…

www.tttech.com



Synchronous Alignment:

Resource Use & Complexity Reduction

Maximize use of network bandwidth and computing resources for

critical embedded functions

• Ensure unambiguous design of key system interfaces

• Reduce uncertainity, jitter and unintended system states (prevents system state

explosion)

Improve functional alignment (and separation!)

• Simplified sensor fusion and distributed processing

• Simplified redundancy management

• Minimize software complexity / simplify functional alignment

Middleware /

Platform

Abstraction

Software

Application

Asynchronous Ethernet

Communication

Synchronous/Asynchronous

Ethernet Communication

Middleware /

Platform Abstraction

Software

ApplicationMiddleware /

Platform

Abstraction

Software

Application

Middleware /

Platform

Abstraction

Software

Application

Middleware /


Software

Application

Middleware /


Software

Application

www.tttech.com



RECEIVERSENDER

MIDDLEWARE

NETWORK

MIDDLEWARE

NETWORK

SYNC LEVEL 1

(NETWORK)

syn

cSYNC LEVEL 2

(MIDDLEWARE)syn

c

SYNC LEVEL 3

(APP. LEVEL)

syn

c

Interface to physical

systems synchronized

(simpified sensor fusion)

Redundancy Mgmt

(Voting)

Comm. Abstraction

Network – Temporal

behavior for all critical

functions defined here!!!

Clean Layered Model: Improved Control of

Latency and Jitter (TTA model)

Interfaces and temporal behavior defined at network level

• Middleware contains parameter-defined communication abstraction and redundancy management (voting)

• Application can handle only functional aspects without temporal interdependencies (no busy waiting, watchdogs, semaphores, …)

• All behavior related to progression of time, not dependant on HW or SW platform

• Supports model-based application design (simple computation tasks!)

• All sensors and actuator access synchronized to µs (using simple IO tasks)

www.tttech.com



Use Cases

www.tttech.com



Launchers

www.tttech.com



Launcher Application – Ariane 6

• Ariane 6 (replacement for Ariane 5) : Planned first flight 2020/2021

• Higher integration levels and SWaP reduction, lower physical complexity

DASIA 2012

www.tttech.com



Launcher Application

Single-fault-tolerance handled in the protocol (network

level) Robust and Highly Reliable Systems

One network configuration – different launcher

configurations Modular embedded platforms

Known latency and minimal jitter for critical

communication Fully deterministic, predictable

WCET in complex integrated systems

Fault-tolerant synchronization Lower software

complexity, predictable operation

Ethernet physical layer 100Base-TX Robust

Seamless integration since the sub-systems are tested

with the flight configuration „Composability“, design

and verification in isolation does not create integration

challenges

Make use of standard Ethernet for development, testing

and operations COTS based

www.tttech.com



Launcher

Application

www.tttech.com



Human Space Flight Application

Up to dual-fault-tolerance handled in the

protocol (network level) Higly reliable

Full determinism (known latency and minimal

jitter) Highly deterministic

Full traffic partitioning (combine platform and

payload) Easy access to shared ressource

e.g. TSP OS / Integrated Architecture

Fault-tolerant synchronization

Seamless integration since the sub-systems

are tested with the flight configuration

Composeability

www.tttech.com



HSF Application – MPCV (Multi

Purpose Crew Vehicle)

October 23rd, 2013 - 7th ADCSS Workshop, “NASA MPCV Use of Ethernet - Time Triggered Gigabit Ethernet on NASA’s

Crewed Exploration Vehicle”, George Eger, LMCO

www.tttech.com



NASA Orion/MPCV 1st Test Flight

(5th December 2014)

www.tttech.com



NASA Orion/MPCV 1st Test Flight

“Congratulation to TTTech! It was a fantastic mission and the TTEthernet Data Network worked perfectly! Thanks for

all of the support over the years…it was great to see it come together and work so incredibly well.”

www.tttech.com



Avionics Networks and System Lifecycle in

Advanced Integrated Systems

www.tttech.com



Satellite Application

Globale synchronized time-base (important for

platform and payload) One trusted timebase

Synchronization to GPS (all data is timestamped with

a precise absolutestamp) One absolute timebase

Full determinism (known age of data important for

platform an payload) Allows distributed real-time

computing

Full traffic partitioning (combine platform and

payload) Easy access to shared ressource e.g.

TSP OS

Seamless integration since the sub-systems are

tested with the flight configuration

Composeability

Real-time Reduced memory needs (no large

buffering SRAM necessary for TTEthernet switches

and embedded systems)

www.tttech.com



Satellite Application

www.tttech.com



Space HW and IP

www.tttech.com



TTEthernet IP and Components

Rad tolerant/hard FPGA

Rad hard ASIC

TTEthernet Space IP Core

TTEthernet Common IP Core

Pegasus(Automotive, Aerospace,

Energy, ...)

TTE-End System Controller Space

ASIC

TTE-End System IP Core Pluto

Space

TTE-End System IP Core Space

TTE-Switch Controller Space

ASIC

TTE-Switch IP Core Space

2014

>2016

www.tttech.com



TTEthernet Products

TTEEnd System A664 Lab

Development Systems

Boards Software

TTESwitch 24 Ports Lab

Support Customization

Integration

Development Hardware & Design Tools Support & Integration & Customization

Test Equipment & Verification Tools Flight and Rugged Products

TTESwitch 3U VPX

Rugged

TTEEnd System A664

Rugged TTEView

TTESwitch A664

A600 Pro

TTEVerify

TTESMC

Chip IP

www.tttech.com



Asynchronous MoCC

www.tttech.com



L-TTA: Asynchronous MoCC (I)

• Resources are reserved for all functions in the system

• It can be proven or assumed that no function will use more

resources than planned (closed system!)

• Strict resource use policing:

• dropping data packets violating temporal boundaries

• exiting non-compliant processes

www.tttech.com



L-TTA: Asynchronous MoCC (II)

Designed to avoid hand-shaking, deadlocks, CMFs

• Periodic processing and sensor sampling

• Defined maximum latency for all data communication

• No temporal interdependancies or synchronism among

computing modules and networking devices

www.tttech.com



Synchronous MoCC

www.tttech.com



TTA: Synchronous MoCC (I)

Can be seen as a special L-TTA case, with network

devices and computing modules in sync

• Fixed latency for all data communication

• Hard RT computing performance

• Distributed (masterless) fault-tolerant system time

www.tttech.com



Deterministic Ethernet

www.tttech.com



Capabilities:

Deterministic Unified Ethernet

What if synchronous links (VLs) are reserved, but the message is not sent?

• … ECU/LRU is not installed

• … Function is currently inactive

Dynamic Bandwidth Release: immediate availability for asynchronous traffic

spacecraft architectures based on...

Documents