Presented at the 31st Real-Time Systems Symposium (RTSS), IEEE, 2010

An Evaluation of SMT-based Schedule SynthesisFor Time-Triggered Multi-Hop Networks

Wilfried SteinerChip IP Design

TTTech Computertechnik AGwilfried.steiner@tttech.com

Abstract—Networks for real-time systems have stringent end-to-end la-

tency and jitter requirements. One cost-efficient way to meet theserequirements is the time-triggered communication paradigmwhich plans the transmission points in time of the frames off-line.This plan prevents contentions of frames on the network and iscalled a time-triggered schedule (tt-schedule).

In general the tt-scheduling is a bin-packing problem, knownto be NP-complete, where the complexity is mostly driven bythe freedom in topology of the network, its associated hard-ware restrictions, and application-imposed constraints. Multi-hop networks, in particular, require the synthesis of path-dependent tt-schedules to maintain full determinism of time-triggered communication from sender to receiver.

Our experiments using the YICES SMT solver show thatthe scheduling problem can be solved by YICES out-of-the-boxfor a few hundred random frame instances on the network.A customized tt-scheduler using YICES as a back-end solverallows to increase this number of frame instances up to tens ofthousands. In terms of scheduling quality, the synthesis producesup to ninety percent maximum utilization on a communicationlink with schedule synthesis times of about half an hour forthe biggest examples we have studied. As a nice side-effect theYICES out-of-the-box approach is immediately applicable forthe verification of existing (even large-scale) tt-schedules and fordebugging more sophisticated tt-schedulers.

I. INTRODUCTION

The types and variants of existing and emerging embeddedreal-time systems are as numerous as the physical processesthat they command and control. For example, the surroundsound system of a home theater has a different set of re-quirements than the flight-management system in an airplane.Naturally, several trade-offs have to be taken into accountwhen developing the embedded real-time system. For real-time scheduling one fundamental trade-off is in the amount ofoff-line versus the amount of on-line computation.

In this paper we address the static scheduling approach[1], [2], [3] in which a complete resource-sharing contractis developed off-line. For time-triggered communication [4],this contract, the tt-schedule, is an offline assignment ofcommunication slots to communication participants. The com-munication participants synchronize their local clocks to eachother and dispatch their messages according the tt-schedule.This way contentions at the shared resource are avoided atdesign time and mutual exclusion is guaranteed.

Time-Triggered Communication comes in various flavors,e.g., the automotive industry is currently adopting FlexRay

[5] for serial production and the avionics industry deploysSAFEbus [6], [7] in the Boeing 777 and TTP [8] in the AirbusA380 and the Boeing 787 aircrafts. While these protocolsare broadcast-based with the intent to operate on physicalbus or hub topologies, Ethernet-like protocols introduce net-work switches that allow concurrent time-triggered dataflows.TTEthernet [9] which has been selected for upcoming spaceprograms is one of the first time-triggered protocols that inher-ently supports concurrent time-triggered communication alsoin multi-hop topologies. Furthermore, as time-synchronizationservices such as IEEE 1588 find their way into standardEthernet equipment it seems only a matter of time whentime-triggered communication shows up also in a “consumer”flavor. As we are looking at these developments in systemsize and complexity, scheduling for time-triggered networksbecomes even more a challenge.

The scheduling problem for a time-triggered system likemany other problems in system design can be formulatedas solutions to systems of constraints. The constraints fordifferent types of problems have different characteristics andspecialized solvers have been developed for each class ofproblems. Recently, a very powerful type of general-purposeconstraint solver has emerged called an SMT solver, a tool thatincorporates decision procedures to solve problems formulatedin first-order logic with respect to background theories. Suchtheories relevant for our problem space are linear equationsand integer theory. Sometimes, an SMT solver can solveindustrial-sized problems of a given class out-of-the-box; inother cases, it is sometimes possible to solve problems that theSMT solver cannot solve on its own by wrapping it in a fairlysimple script. Examples include [10], [11], [12]. The benefitof doing this is simplicity, flexibility, and division of labor:a substantial community, spurred by annual competitions, iscontinually improving the performance of SMT solvers. Eventhough a solver customized to a particular class of problems,or an SMT solver with a customized inner strategy, mightoutperform a standard SMT solver wrapped in a script, thedevelopment cost might outweigh the advantage, and contin-ued development would be needed to maintain that advantage.In this paper, we develop a strategy for solving the schedulingproblem for time-triggered systems using SMT solvers andreport on its performance.

In general the scheduling of a distributed real-time sys-tem requires a holistic system view that takes into account

1

hardware resources in the end systems and in the networkdevices as well as task-level execution times and their inter-dependencies. Ideally, all these system parameters are ad-dressed at once to find an optimal resource utilization. How-ever, with growing system size and complexity the holis-tic scheduling approach becomes more and more difficultand for certain industrial-sized systems (having hundreds oftasks and thousands of messages) almost infeasible. Industrybest-practice solutions to this dilemma are typically multi-staged scheduling approaches: application-level schedules andcommunication schedules can be developed independentlyto some degree and composed afterwards. Should resourceconflicts in the composition be detected, these conflicts canbe propagated back to the individual schedulers as additionalconstraints. In this paper we, therefore, focus on the synthesisand verification of communication schedules for time-triggeredcommunication networks and assume a fixed task assignmentto the communication participants. Still, we allow arbitrarytask dependencies resulting in frame transmission dependen-cies for the schedule synthesis and verification. These taskdependencies are formulated as dependency graphs in analogyto the task graphs [3] and are intended as a generic interfaceto task-level schedulers.

The main contribution of this paper is the formal specifica-tion of the scheduling constraints for time-triggered multi-hopnetworks and the study of the respective schedule synthesisand verification. We evaluate the performance of the state-of-the-art SMT solver YICES [13] in the schedule synthesis usingthe defined formal schedule constraints and show that YICESout-of-the-box is capable of scheduling a TTEthernet networkwith hundreds of frame instances. An advanced tt-scheduleris discussed that uses the C-API to run YICES as a back-endsolver and as a result increases the number of frame instancesthat can be scheduled by at least one order of magnitude. Asanother contribution, our experiments serve as case study onhow to use the YICES C-API.

A reflection on the last decades of real-time scheduling isgiven in [14] with a focus on dynamic scheduling. Differentattempts have been proposed to introduce flexibility into thestatic time-triggered paradigm, for example “pre-scheduling”as introduced in [15]. In this paper, we are concerned withstatic scheduling which has been developed and is studiedin various groups [1], [2], [3], [16]. We do not claim a newscheduling algorithm per se, but rather discuss the appropriate-ness of a general purpose tool for static scheduling problems,in particular how to use an SMT solver for static scheduling.In this respect our work is in the spirit of [17] and [18]. [17]discusses the application of a SAT solver for the scheduling ofa distributed system, though the task set is limited to about fiftytasks. In [18] the application of a constraint solver is discussedin the scheduling of a task set of hundreds of tasks, thoughthe network size is limited to four nodes. Our work extendsthese results from two perspectives: first we are focusing onnetwork-centric designs which exceed the networks in thebefore mentioned publications significantly. Secondly, we usea different class of tools. We also use automatized tests for the

performance evaluation of our approach whereas the previouswork focused on some example use cases.

Voss et al. [19] use the model-checker SAL [20] in theschedule synthesis for time-triggered schedules. While thenumber of tasks to be scheduled is rather limited it has thebenefit that the SAL description is usable as a formal specifica-tion of the actual distributed algorithms. This potentially alsoallows the verification of general properties. This beauty is lostby “brute-force” encoding of the constraints in YICES as wedo for the sake of increasing the scheduling set. Schedulingcan also be seen as a special case of planning as mostly done inthe Artificial Intelligence community. One of the first activitiesrelating SAT solving to planning problems is reported in [21].

We introduce the basic formalism and concepts in Section IIfollowed by the formal schedule constraints in Section III. Wethen discuss the SMT-based scheduling approach in SectionIV and present parts of the C-code of the basic and advancedtt-scheduler. The generic testbench and synthesis results aswell as performance discussions are presented in Section V.Finally, we conclude in Section VI.

II. BASIC TERMINOLOGY AND CONCEPTS

Formally the physical topology of a network is defined by aundirected graph G(V,E), where end systems and switches arevertices V and the physical communication links connectingvertices are edges E. Figure 1 depicts an example of anetwork topology with eight vertices (six end systems andtwo switches).

A

B CH

G

F

D

E

Physical Topology Dataflow Path Virtual Link

Fig. 1: Example of a TTEthernet network with six end systemsand two switches connected in multi-hop

In this paper we focus on bi-directional physical com-munication links. Hence, each physical communication linkconnecting two vertices defines two directed “dataflow links”.We denote the set of dataflow links by L:

∀v1, v2 ∈ V : (v1, v2) ∈ E ⇒ [v1, v2] ∈ L, [v2, v1] ∈ L (1)

where “(x, y)” denotes an unordered tuple and “[x, y]’denotes an ordered tuple.

A sequence of dataflow links li is said to form a “dataflowpath”. The dataflow path in a network is represented by adirected path connecting one vertex (the sender) with exactlyone other vertex (the receiver). An example of a dataflow pathfrom A to F is depicted by the dotted line in Figure 1.

2

We denote the set of dataflow paths by DP and formallyexpress a dataflow path p from a sender v1 to a receiver vrby the sequence of its dataflow links:

p = [[v1, v2], ..., [v(r−1), vr]] (2)

where the shortest dataflow path has length one (i.e.p = [v1, vr]).

A dataflow path defines, thus, a route from a sender toexactly one receiver. Information between the sender andreceiver is communicated in the form of messages that we call“frames” according to the Ethernet convention. We denote theset of all frames by F with fi ∈ F and express the associationof a frame fi with a respective dataflow path p via its dataflowlinks: the instance of frame fi on a dataflow link [v1, v2] isdenoted by f [v1,v2]i .

Frames may be delivered from a sender to multiple re-ceivers. Therefore, we say the individual dataflow paths be-tween the sender and each single receiver together form a“virtual link” (according the ARINC 664-p7 definition). Wedenote the set of virtual links by VL. Formally, a virtual linkvl is expressed by the union of its dataflow paths: vl =

⋃pi.

Furthermore, we are interested in well-formed virtual linksonly: any two vertices va, vb ∈ V that are part of a dataflowpath pi ∈ vl and are also part of another dataflow path pj ∈ vlare linked by the same sequence of dataflow links. Hence, avirtual link defines a directed tree structure having the senderas root and the receivers as leaves. For each frame fi ∈ F ,we formally denote the root by first(fi) (i.e. the first dataflowlink in the virtual link of fi) and, likewise, we denote the setof leafs by last(fi) (i.e. the set of last dataflow links in thevirtual link of fi). The problem of finding such tree structuresin the network is already well-studied in the context of Steinertrees and therefore outside the scope of this paper. Hence, weassume that all virtual links are a priori given and specified interms of their individual dataflow paths.

A TT frame fi on a dataflow link [vk, vl], f[vk,vl]i , is fully

temporally specified by the following triple (as defined by theTime-Triggered Architecture [4]):

f[vk,vl]i = {fi.period , f

[vk,vl]i .offset , fi.length} (3)

The TT frame period and frame length are given a priori; itis the task of the tt-scheduler to assign values to f [vk,vl]i .offsetfor all frames fi ∈ F on all dataflow links [vk, vl] ∈ L in thenetwork.

III. FORMAL TIME-TRIGGERED SCHEDULINGCONSTRAINTS

This section defines the constraints on f[vk,vl]i .offset that

apply in a time-triggered network.

A. Contention-Free Constraints

The most basic constraints in time-triggered schedulingaddress mutual-exclusion on the dataflow links, hence, theyguarantee contention-freedom. This means that a sender orrelaying vertex will dispatch, or respectively relay, a frame

only after the previous frame has been processed. In a puretime-triggered network, the term “processed” refers to thecompleted delivery of the previous frame. In a mixed time-triggered / event-triggered network, the term processed can berelaxed as the previous time-triggered frame may get delayedby an event-triggered frame in transition. In this paper weaddress only pure time-triggered networks.

The following constraints describe the case of a single frameperiod assigned to all frames in the network:

∀[vk, vl] ∈ L,∀fi, fj ∈ F :

((fi 6= fj) ∧ ∃f [vk,vl]i ∧ ∃f [vk,vl]j )⇒

((f[vk,vl]i .offset ≥ f [vk,vl]j .offset + fj .length)

∨(f[vk,vl]j .offset ≥ f [vk,vl]

i .offset + fi.length)) (4)

In case of varying, but harmonic frame periods Equation 4has to be generalized as follows:

∀[vk, vl] ∈ L,∀fi, fj ∈ F,

∀a ∈ [0..

(LCM (F.period)

fi.period− 1

)],

∀b ∈ [0..

(LCM (F.period)

fj .period− 1

)] :

((fi 6= fj) ∧ ∃f [vk,vl]i ∧ ∃f [vk,vl]j )⇒

((a× fi.period) + f[vk,vl]i .offset ≥

(b× fj .period) + f[vk,vl]j .offset + fj .length) ∨

((b× fj .period) + f[vk,vl]j .offset ≥

(a× fi.period) + f[vk,vl]i .offset + fi.length)) (5)

where LCM (F.period) denotes the least common multipleof all frame periods and is called the “cluster cycle”. Thecluster cycle is an analogous concept to the “hyper period” instatic scheduling introduced as early as in [1].

One way to reduce scheduling complexity (as discussed in[22]) is the introduction of a “raster” that splits the clustercycle in equally sized “slots”. Assuming one frame per slot,Equation 5 is simplified to the following:

∀[vk, vl] ∈ L,∀fi, fj ∈ F,

∀a ∈ [0..

(LCM (F.period)

fi.period− 1

)],

∀b ∈ [0..

(LCM (F.period)

fj .period− 1

)] :

((fi 6= fj) ∧ ∃f [vk,vl]i ∧ ∃f [vk,vl]

j )⇒

((a× fi.period) + f[vk,vl]i .offset 6=

(b× fj .period) + f[vk,vl]j .offset) (6)

We will use this slot mechanism in our scheduling exper-iments. Note, the frame offset is then identified by a slotnumber in the communication schedule.

3

B. Path-Dependent Constraints

The path-dependent constraints describe the requirementthat within the dataflow path of a frame fi the dispatch pointsin time on two adjacent dataflow links will be well-timed. Thismeans that the dispatch point in time on a succeeding dataflowlink will be scheduled only after it has been received from thepreceding dataflow link.

Let p be the dataflow path of a frame fi from its originalsender v1 to the final receiver vr. Then the path-dependentconstraints for the dataflow path p are given by:

∀vl ∈ VL : ∀pi ∈ vl : ∀[vx, vj ], [vj , vy] ∈ pi :

(f[vj ,vy ]i .offset)− (f

[vx,vj ]i .offset)

≥ max (hopdelay) (7)

where max (hopdelay) is an off-line configurable upperbound on the maximum latency over a single hop and[vx, vj ], [vj , vy] denote any two adjacent dataflow links in thedataflow path p connected through a relaying vertex vj .

C. Bounded Switch Memory Constraints

Analogously to the path-dependent constraints the hardwareparameters (i.e. memory size, buffer depth, etc.) in the verticesform a set of constraints on the maximum delay time withina relaying vertex vj .

∀vl ∈ VL : ∀pi ∈ vl : ∀[vx, vj ], [vj , vy] ∈ pi :

(f[vj ,vy ]i .offset)− (f

[vx,vj ]i .offset)

≤ membound (8)

where membound is a constant driven by said hardwareparameters. This simple formulation of switch-memory con-straints as a bound on the relay delays of the individual framesis a property of time-triggered communication. In contrastto unsynchronized communication, here we do not have toconsider contentions on the outgoing ports of a switch whichrequires careful queuing analysis.

D. Simultaneous Relay Constraints

Though not a conceptual constraint, there is an optionalimplementation-driven constraint to dispatch a frame fi simul-taneously on all outgoing dataflow links of a relaying vertexvj within the virtual link vl.

∀vl ∈ VL : ∀pk, pl ∈ vl :

∀[vj , vb] ∈ pk,∀[vj , vd] ∈ pl :

(f[vj ,vb]i .offset) = (f

[vj ,vd]i .offset) (9)

E. End-to-End Transmission Constraints

The end-to-end transmission constraints describe the maxi-mum allowed end-to-end latency for a frame fi according to anapplication’s requirements. In case of a multicast transmissionwe assume that the bounds specified will be the same for allreceivers.

∀fi ∈ F : ∀[vl, vk] ∈ last(fi) :

f[vl,vk]i .offset − ffirst(fi )

i < max(latency) (10)

F. Application-Level Constraints

Application-level constraints on the communication sched-ule arise for example from task-level dependencies. Theseconstraints are typically of the form ”frame fa shall bedispatched at least ∆ msec before frame fb”:

first(fa) = [vl, vk] ∧ first(fb) = [vm, vn] :

f [vl,vk]a .offset + ∆ ≤ f [vm,vn]b (11)

Upper bounds on the relation of two dispatch points in timecan be specified analogously.

The application-level constraints specify the relative in-put/output behavior of a component. Hence, they can be useddirectly to model or even derive aspects of a task schedule.

G. Other Constraints

1) Protocol-Control-Flow Constraints: Time-triggeredcommunication requires the synchronization of the localclocks of the communication participants. This is typicallyachieved by the exchange of clock synchronization messages.While SAFEbus uses dedicated communication links forsynchronization, newer time-triggered protocols such as TTP,FlexRay, or TTEthernet use a single physical network for bothdataflow and synchronization. TTP and FlexRay in particularuse regular frames also for synchronization, such that noadditional constraints are necessary in principle. However, itmay be a design decision selecting specific offset values forthe synchronization messages. This can be achieved via theapplication-level constraints as discussed above.

2) Domain-Specific Constraints: Domain-specific con-straints can be used to relate not only two frames to eachother, as application-level constraints, but to express system-atic relation within a group of frames. Audio and video stream,for example, have certain characteristics that can be expressedin a generic way.

H. Example Visualization of Constraints

Figure 2 depicts an example schedule of a network in“snowflake” topology (see Figure 6.c) consisting of 50dataflow links and 100 frames, resulting in about 1000 sched-uled frame instances. The x-axis depicts the identities of thedataflow links, the y-axis depicts the slot position in theschedule (we assume a uniform frame period). A dot in thediagram depicts the assigned slot position of a frame instanceon the respective dataflow link.

Arrows depict the application-level constraints (Equations11), e.g. the arrow from (8, 7) to (24, 50) represents followingconstraint: f8a .offset = 7, f24b .offset = 50, ∆ = 43 ⇒ 7 +43 ≤ 50. Likewise lines depict dependencies between frameinstances as generated by the constraints Equations 7, 8, 9.

4

0

20

40

60

80

100

120

140

0 5 10 15 20 25 30 35 40 45 50

Slo

t Pos

ition

in S

ched

ule

[inde

x]

Dataflow Links [index]

Fig. 2: Example tt-schedule; lines and arrows represent de-pendencies. The diagram is generated automatically from thesynthesized offset data using gnuplot. Arrows and lineshave been added manually.

IV. SMT-BASED SCHEDULING APPROACH

Each SMT-based scheduling approach requires two phases:(a) generation of scheduling constraints and adding them to the“logic context” of the solver and (b) invocation of the solver.The design freedom for a tt-scheduler is, thus, determinedby the interrelation of these phases. For our experimentswe developed two tt-schedulers: a basic scheduler and anincremental scheduler. In the basic scheduler all scheduleconstraints are generated upfront, then the SMT solver is calledexactly once. The incremental scheduler frequently alternatesphase (a) and (b) and uses backtracking features of YICES.

GenericSchedule

Constraints

Schedule Synthesis

Schedule Verification

GenericSchedule

Constraints

Schedule Synthesis

Schedule Verification

Schedule Strategies

SpecificConstraints

SpecificConstraints

(a) Basic Scheduler (b) Advanced Modular Scheduler

Dataflow Specification TT Schedule

TT ScheduleDataflow Verification

Dataflow Specification TT Schedule

TT ScheduleDataflow Verification

Fig. 3: Schematic overview of a modularized tt-scheduler

As depicted in Figure 3, a SMT-based tt-scheduler can beused both, for the synthesis of a schedule as well as for itsverification.

A. Basic TT-Scheduler

The basic tt-scheduler generates the scheduling constraintsintroduced in Section III for all frames and adds them to thelogical context of the SMT solver. Then the SMT solver isinvoked to check whether the logical context is consistent.When the SMT solver returns successfully (l_true) thenwe know that the schedule constraints are satisfied and a

valid variable assignment (also called a “model”) for thislogical context can be accessed through the appropriate C-API function (yices_get_model()). This model is the tt-schedule. The C-code of the basic tt-scheduler is depicted inListing 1.

Listing 1: Basic Scheduler - Pseudo C-Codepublic void basic_scheduler() {for (i=1; i<MAX_NUMBER_FRAMES+1; i++){make_constraints(i); }

if (yices_check(ctx) == l_true){model = yices_get_model(); }

}

B. Incremental TT-Scheduler

In order to exceed the size of the logic context that aSMT solver is able to solve directly we have developedan incremental tt-scheduler. Here, we do not generate theschedule constraints for all frames at once, but only for asmall subset at a time and invoke the SMT solver then. Eachtime the solver returns successfully, the partial tt-schedule isadded explicitly to the logical context, hence the positions ofthe frames of the current subset are fixed in the tt-schedule. Incase that the solver returns an inconsistency, we backtrack andincrease the size of the subset of frames to be scheduled. Whenthe SMT solver does not terminate within a given timeout, orthe size of the subset grows beyond a defined threshold weabort the scheduling approach. The incremental tt-scheduler isdepicted in Listing 2 (abort conditions are not listed).

Listing 2: Incremental Scheduler - Pseudo C-Codepublic void inc_scheduler() {

int head = 1;int tail = SMT_FEED;int bt_ptr = 0;int bt_stack[MAX_NUMBER_FRAMES+1];yices_context ctx = yices_mk_context();yices_model model;

while (tail < MAX_NUMBER_FRAMES+1) {make_constraints(head, tail);if (yices_check(ctx) == l_true) {model = yices_get_model();for (i=head; i<tail+1; i++) {for (j=head; j<MAX_LINKS+1; j++) {yices_get_int_value(model,

y_offset_decl[order[i]][j],&offset[order[i]][j]);

}}(void) place(head, tail);yices_push(ctx);

bt_stack[bt_ptr] = tail;bt_ptr++;bt_start = 1;head = tail + 1;tail = tail + stepsize;

}else { // backtrack

if (bt_start == 1) {yices_pop(ctx);bt_ptr--;bt_stack[bt_ptr] = 0;bt_count++;

}yices_pop(ctx);bt_ptr--;head = bt_stack[bt_ptr] + 1;bt_stack[bt_ptr] = 0;bt_start = bt_start++;

}}

5

The current range of frames to be scheduled is definedby head and tail. SMT_FEED defines the default numberof frames for which scheduling constraints are generatedconcurrently. We generate the constraints for this range offrames (make_constraints()) and check whether the logiccontext is satisfiable (yices_check(ctx) == l_true).If so, then we ask YICES for a variable assignment (amodel) that satisfies the logic context (yices_get_model())and read the variable assignments into the local arrayoffset[frame_id][link_id]. We then explicitly add thesevariable assignments as constraints to the logic context(place()). Furthermore, we store the index (tail) up towhich the frames have successfully been scheduled on a stackbt_stack[bt_ptr].

The assignments of the variables to fixed values is stronglydependent on the scheduling sequence of the frames. Inparticular, we may assign offset values to frame instanceswhich potentially violate the constraints of frames that will bescheduled later on. There are two reasons for a violation: (a) anapplication-level conflict or (b) the current frame does not fit asa result from inefficient scheduling of the previous frames. Weaddress this issue by backtracking: whenever YICES identifiesan inconsistent logic context we freeze tail and move head

towards the beginning of the frame sequence. Thereby wegradually remove previously variable assignments and increasethe group size of frames to be scheduled. For that we usethe backtracking function of YICES yices_pop(ctx) andour backtracking stack bt_stack[bt_ptr] as depicted in thelisting.

As the schedule sequence is essential, the tt-scheduler takesthe order of this sequence as a parameter: order[] definesthe sequence in which the frames are processed. This allowsthe implementation of different scheduling strategies.

In the worst case the backtracking approach leads tobacktracking to the first frame scheduled in which case theincremental tt-scheduler has no benefit over the basic tt-scheduler. Therefore, when the tt-scheduler does not find aschedule within reasonable time, we can abort the synthesisprocess, backtrack to the latest consistent logic context, andreturn the schedule developed so far. Listing 3 depicts theYICES abort function (yices_interrupt(ctx)) triggeredby a system call.

Listing 3: Signal Handler - Pseudo C-Codesignal(SIGALRM, sigint_handler);void sigint_handler(int signum) {

yices_interrupt(ctx);timeout = 1; }

C. Schedule Correctness and Dual-Use Schedulers

The “dual-use” property of SMT-based tt-schedulers is veryvaluable in the development and verification of complex tt-schedulers as a schedule cannot be manually verified once itexceeds about a hundred frame instances. Figure 4 gives anexample output of a tt-scheduler consisting of assigned slotpositions for about nine thousand frame instances. Clearly, the

manual verification of interdependencies as depicted in Figure2 becomes infeasible.

0

100

200

300

400

500

600

700

800

0 5 10 15 20 25 30 35 40 45 50

Slo

t Pos

ition

in S

ched

ule

[inde

x]

Dataflow Links [index]

Fig. 4: Visual Inspection Limitation

As discussed above, the basic tt-scheduler realizes a simplefunction for generation of scheduling constraints, which canbe proven correct by informal arguments and accompanyingtestcases. Hence, as long as the SMT solver is correct, thebasic tt-scheduler will produce correct tt-schedules. To achieveeven more diversity, different SMT solvers can be used forsynthesis and verification.

More complex tt-schedulers, like the incremental schedulerdiscussed above, involve more sophisticated programmingconstructs and their design is therefore error-prone. Indeed,a particular bug discovered during the development of theincremental scheduler led to unsynchronized backtracking ofYICES and the local backtracking stack bt_stack[bt_ptr].As a result the schedule constraints for some frames were notgenerated under certain backtracking scenarios. We thereforemodified the basic tt-scheduler to operate as a verificationtool: in addition to all constraints discussed so far we alsogenerate all offset assignments as produced by the incrementaltt-scheduler as constraints. The basic tt-scheduler checks thenfor inconsistencies in the logic context and we derive a proofof schedule correctness when no inconsistencies are reported.

V. SYNTHESIS PROCEDURE AND RESULTS

A. Test Bench and Benchmark Description

In order to evaluate the performance and quality of ourYICES-based tt-schedulers we have developed a generic test-bench operating in three steps as depicted in Figure 5. In afirst step we create topologies, in the second step we definevirtual link and frame assignments, and in the third step wedefine application-level dependencies.

1) Topologies: We tested three topologies: star, tree, and“snowflake” depicted below. While the star and the treetopologies are typical for federated system architecture, thesnowflake topology may be more applicable to an integratedarchitecture. In this case the leaves of the topology represent

6

2

1 4

6

9

1110

3

87

A

B

C

D

E

A

B

C

D

E

(a) Physical Link Generation (b) Virtual Link Generation (c) Application Constraints Generation

Fig. 5: Testbench for scheduling performance assessment

partitions within a single physical end system. Also, thesnowflake topology is similar to the “fat tree” architecturecommon in data centers.

(a) Star Topology (b) Tree Topology (c) Snowflake Topology

Fig. 6: Topologies under test

2) Message Assignments: Message assignment is done bygenerating virtual links each consisting of one sender and aset of receivers. The sender is randomly selected from the setof dataflow links not having a predecessor. The selection ofreceivers is a configurable function of the sender: each senderbelongs either to (a) the broadcast group (b) multicast group,or (c) the unicast group. A sender of the broadcast groupis allowed to randomly select the set of receivers from allpossible receivers (those dataflow links not having a successordataflow link); a sender of the multicast group is allowedto transmit only to a subset of configurable group size ofreceivers; the unicast group is a special type of multicastwith group size two. This configurable function allows us tobuild more accurate communication scenarios as it is unlikelythat each vertex will communicate with each other vertex ina given system. Our experiments cover end-to-end commu-nication between leaf vertices in the topology. On demand,the testbench can be easily extended also to cover dataflowbetween intermediate vertices.

In our experiments we scheduled two distributions for eachtopology: in the high (HI) distribution all senders belong to thebroadcast group, in the low (LW) distribution between a thirdand a half of the senders belong to the broadcast group andmost of the remaining senders belong to one or two multicastgroups as depicted in Table I.

We consider a uniform period for all frames to be scheduled.The testbench can, though, be easily extended to harmonicframe periods. Non-harmonic frame periods can be accountedfor by offline partitioning of the network bandwidth and theindividual synthesis of a schedule for each set of non-harmonicperiods.

Topology Broadcast Multicast Unicaststar 8 12 -tree 8 6 2

snowflake 8 12 -

TABLE I: Sender to group assignment in the low-distributionscenarios; group size for multicast is 6

3) Application-Level Constraints: The testbench generatesapplication-level constraints in two steps: first the framesare grouped in to “application groups” of configurable size,secondly a precedence relation between the frames withinthe application group is established. The precedence relationgenerates a random tree dependency within an applicationgroup. In our experiments we selected an application groupsize of five frames with a random distribution in the temporalprecedence in [0; 50] communication slots resulting in maxi-mum temporal distances of 2% to 50% of the cluster cyclesunder test. An example of such a precedence relation is givenin Figure 2 by the tree structure consisting of the arrows.

B. Synthesis Results

The synthesis runs were executed on a Xeon CPU at 2GHzand we have observed similar execution times on a regulardesktop CPU. The main driver for our synthesis environmentwas the memory consumption growing up to ten GByte ofRAM.

The performance of the basic tt-scheduler is depicted inFigure 7. Here a star topology with twenty senders, belongingto the broadcast group, is depicted.

0

200

400

600

800

1000

1200

1400

1600

1800

20 40 60 80 100 120 140 160 180 200

200"exec_negative.plt" u 1:2

Fig. 7: Synthesis times of the basic tt-scheduler

The x-axis represents the number of frames to be scheduled;the y-axis the schedule synthesis time in seconds. As depictedthe synthesis times are reasonable up to about 120 framesto be scheduled. Beyond that the synthesis times grow quitesignificantly; scheduling attempts have been aborted after 30minutes depicted as “+” in the graph.

7

In the following we discuss schedule synthesis times ofthe incremental tt-scheduler. Figures 8, 9, and 10 depict theschedule synthesis times versus the number of frames andframe instances to be scheduled for the different topologies.The x-axis depicts the number of frames to be scheduled, theleft y-axis depicts the synthesis times in minutes and the righty-axis shows the number of frame instances in thousands (oneframe is transported over multiple dataflow links, a frameinstance is the frame with respect to a dataflow link). Asdiscussed, two distributions are depicted for each topology:a high (HI) distribution and a low (LW) distribution. Forthe path-dependent constraints we set max (hopdelay) = 1,meaning that the offset expressed in slots of a succeedingdataflow link must be at least one higher than in its precedingdataflow link. For the bounded switch memory constraintswe set membound = 4, meaning that a frame resides ata maximum for four slots within the switch memory. Thesimultaneous-relay constraints are asserted in all scenarios.The end-to-end transmission constraints are set to high enoughbounds such that they will allow the maximum switch memorybound delay in each switch.

Each dot in the figures represents a schedule synthesisof a new randomly generated frame set. Hence, each figurepresents twenty random frame distributions with increasingframe number from one hundred to one thousand.

0

2

4

6

8

10

12

14

16

18

20

0 200 400 600 800 1000 0

1

2

3

4

5

6

7

8

9

10

11

Syn

thes

is T

ime

[min

]

Fra

me

Inst

ance

s [1

03 ]

Frames [num]

Synth. Time HIFrame Inst. HI

Synth. Time LWFrame Inst. LW

Fig. 8: Synthesis Times Star Topology

As depicted in Figures 8, 9, 10 the schedule synthesistimes are largely independent from the topology, though, thetree topology shows a slight increase which is most likely aconsequence of the higher number of frame instances as wellas path lengths. The longest schedule synthesis times are abouthalf an hour which meets industrial development processesvery well. In fact, it is fast enough such that alternatives inthe frame assignments can be tested very easily.

Figure 11 and Figure 12 discuss the quality of the synthe-sized schedules. Figure 11 depicts the minimum number ofslots required to for the six scenarios.

Not surprisingly, the scenarios with high utilization (havingsenders of the broadcast group only) demand the most slotsin the communication schedule. We also see that the relative

0

5

10

15

20

25

30

35

0 200 400 600 800 1000 0

2

4

6

8

10

12

14

16

18

20

Syn

thes

is T

ime

[min

]

Fra

me

Inst

ance

s [1

03 ]

Frames [num]

Synth. Time HIFrame Inst. HI

Synth. Time LWFrame Inst. LW

Fig. 9: Synthesis Times Tree Topology

0

5

10

15

20

25

30

35

0 200 400 600 800 1000 0

2

4

6

8

10

12

14

16

Syn

thes

is T

ime

[min

]

Fra

me

Inst

ance

s [1

03 ]

Frames [num]

Synth. Time HIFrame Inst. HI

Synth. Time LWFrame Inst. LW

Fig. 10: Synthesis Times Snowflake Topology

100

200

300

400

500

600

700

800

900

1000

1100

0 200 400 600 800 1000

Slo

ts in

Sch

edul

e [n

um]

Frames [num]

Star HIStar LWTree HI

Tree LWSnowflake HI

Snowflake LW

Fig. 11: Scheduling Quality: Number of Slots required

utilization grows with the minimum number of slots required.We account for this behavior as the result of the application-level constraints that may cause a chain of frame dispatchesas up to 4× 50 = 200 slots in the worst case (e.g., see Figure2 depicting a chain of about 140 slots in length).

8

Figure 12 depicts the utilization in percent relative to theminimum number of slots required as shown in Figure 11. Forexample: in the Snowflake LW scenario at 1000 frames, thereexists a dataflow link which is utilized over 90% (hence, morethan 900 slots are occupied on this dataflow link).

20

30

40

50

60

70

80

90

100

0 200 400 600 800 1000

Max

imum

Util

izat

ion

[%]

Frames [num]

Star HIStar LWTree HI

Tree LWSnowflake HI

Snowflake LW

Fig. 12: Scheduling Quality: Maximum Utilized Dataflow Link

As depicted, the HI scenarios have less relative utilizationthan the LW scenarios. Especially the Star HI scenario pro-vides utilization of only just above 50%. Although this isstill acceptable compared to a utilization level usually metby unsynchronized traffic, we are interested in improvingthe schedule quality by means of changing the hardwareparameters: increasing memory bounds and disabling simul-taneous relay. Figure 13 shows the significant impact of thesimultaneous-relay constraints. By removing the simultaneousrelay requirements we almost double the scheduling perfor-mance. On the other hand, the increase in memory in theswitches has relative little impact.

35

40

45

50

55

60

65

70

75

80

85

90

0 200 400 600 800 1000

Util

izat

ion

[%]

Frames [num]

Mem 4Mem 8

Mem 16Mem 4, Simul.OffMem 8, Simul.Off

Mem 16, Simul.Off

Fig. 13: Relative Utilization of the Star HI scenario under dif-ferent Switch Memory Bound Constraints and Simultaneous-Relay Constraints

The scheduling problems consist of about 106 to 107

constraints. Figure 14 shows constraint generation time forschedule verification. This is only the time it takes to generatethe constraints and feed them into YICES. As depicted themaximum time is well below five minutes. The actual verifi-cation times are negligible being well below ten seconds.

0

50

100

150

200

250

300

350

400

0 200 400 600 800 1000

Con

stra

ints

Gen

erat

ion

Tim

e [s

ec]

Frames [num]

Star HIStar LWTree HI

Tree LWSnowflake HI

Snowflake LW

Fig. 14: Verification Performance of the basic tt-scheduler

C. Real-Time Performance Discussion

The experiments discussed use a raster scheme, that splits upreal-time into slots which are then assigned to frame instances.In this section we discuss how the raster scheme transformsto real-time numbers based on communication properties asgiven by Ethernet.

Ethernet has defined minimum and maximum frame lengthsof 84 and 1542 octets, respectively (already including 12octets inter-frame gap). Typical wire-speeds for Ethernet are100 Mbit/sec (ARINC 664-p7), 1 Gbit/sec (TTEthernet) upto several tens of Gbit/sec in data-centers. Hence, for 100Mbit/sec we derive a frame length of [6.72µsec; 123.36µsec]and for 1 Gbit/sec of [0.672µsec; 12.336µsec].

Another important factor in a time-triggered communicationprotocol is the maximum deviation of any two correct localclocks in the system. This is called the precision of the systemand a typical industrial value (considering state-of-the artoscillators) is about 1µsec and even below.

In a store-and-forward switch the window size is the max-imum frame length plus the precision in the system, as ithas to be guaranteed that a switch will not start relayinga frame before it was completely received. Note, that thesenumbers do not include wire-imposed latency which has tobe accounted for in geographically large networks or internalhardware overheads. Still the given intervals represent thecorrect order of magnitude.

Given a store-and-forward switch and a good precision of0.5µsec we derive slot size of [7.22µsec; 123.86µsec] and[1.172µsec; 12.836µsec]. In a cluster cycle consisting of 1000slots we get cycle lengths of [7.22msec; 123.866msec] and[1.172msec; 12.836msec]. End-to-end latency is driven by thenumber of dataflow links in the frame path, the slot size, and

9

the storage time in the switch memories which is bound bymembound slots in our framework. The end-to-end latencyis then given by (path.length − 1) × membound . Assumingmembound = 4 and for path lengths of [2; 8] dataflow linkswe derive [4; 28] slots maximum end-to-end latency. Thisresults in an achievable end-to-end latency in a 1 Gbit/secsystem of: 4.688µsec in star topology, 14.064µsec in thesnowflake topology, and 32.816µsec in the tree topology.

A distinguished aspect of time-triggered communication isthat the communication jitter is always bound by the precisionin the system. Hence, in the examples above the jitter is always0.5µsec.

VI. CONCLUSION

The growing size of time-triggered networks and theirincreasing complexity demand high-performance schedulers toplan the communication schedule off-line. While traditionallyspecialized tools have been developed for static scheduling wehave discussed the usage of a general-purpose tool and havedemonstrated its applicability to industrial-sized networks. Weintroduced the SMT solver YICES as back-end solver foran incremental tt-scheduler and YICES out-of-the-box forschedule verification.

The scheduling speed and performance allows testing alter-natives and design choices. Indeed we were able to identifythe simultaneous-relay constraints as scheduling bottleneckand have shown a significant increase in utilization whensimultaneous dispatch of a frame instance on all outgoingdataflow links of a switch is not required.

We tested several topologies under different network loadsand demonstrated that YICES is able to schedule moderate-sized problems on its own while a scheduler using YICESas backend solver is even capable of scheduling large-scalenetworks of embedded real-time systems. Due to the noveltyof time-triggered multi-hop communication we have analyzedsynthetic testcases only. On the other hand the presentedapproach demonstrates that such networks can be configuredby the use of standard tools. Hence, SMT-based tt-schedulingmay be an enabling technology that fosters the industrialadoption of time-triggered multi-hop networks.

In this paper we have defined the formal scheduling con-straints that apply in a time-triggered multi-hop network.These constraints precisely formulate the scheduling problemand in future work we will compare the scheduling per-formance of the presented generic approach with industrialtools currently under development. Other future work is theextension of the presented approach towards a broader setof hardware properties as for example different wire-speeds.A particular follow-on research to the one reported herein isconcerned with integrating dynamic dataflows within the sameframework.

ACKNOWLEDGMENTS

The research leading to these results has received fundingfrom the European Community’s Seventh Framework Pro-gramme (FP7/2007-2013) under grant agreement n◦236701

(CoMMiCS). CoMMiCS is hosted by the Computer ScienceLaboratory at SRI International where this work has beenperformed.

REFERENCES

[1] T. P. Baker and A. Shaw, “The cyclic executive model and Ada,” TheJournal of Real-Time Systems, vol. 1, pp. 120–129, 1989.

[2] H. Kopetz, Real-Time Systems. Kluwer Academic Publishers, 1997.[3] D.-T. Peng, K. G. Shin, and T. F. Abdelzaher, “Assignment and schedul-

ing communicating periodic tasks in distributed real-time systems,”IEEE Trans. Softw. Eng., vol. 23, no. 12, pp. 745–758, 1997.

[4] H. Kopetz and G. Bauer, “The time-triggered architecture,” Proceedingsof the IEEE, vol. 91, no. 1, pp. 112 – 126, Jan. 2003.

[5] R. Mores, G. Hay, R. Belschner, J. Berwanger, C. Ebner, S. Fluher,E. Fuchs, B. Hedenetz, W. Kuffner, A. Krger, P. Lohrmann, D. Millinger,M. Peller, J. Ruh, A. Schedl, and M. Sprachmann, “Flexray thecommunication system for advanced automotive control systems,” InProceedings of SAE 2001 World Congress, 2001.

[6] H. Hoyme and K. Driscoll, “Safebus (tm),” IEEE Aerospace andElectronics Systems Magazine, vol. 8, no. 3, pp. 34–39, March 1993.

[7] T. Carpenter, K. Driscoll, K. Hoyme, and J. Carciofini, “Arinc 659scheduling: problem definition,” in Proceeding of Real-Time SystemsSymposium, Dec. 1994, pp. 165–169.

[8] H. Kopetz, TTP/C Protocol – Version 1.0. Vienna, Austria: TTTechComputertechnik AG, Jul. 2002, Available at http://www.ttagroup.org.

[9] W. Steiner, TTEthernet Specification, TTA Group, 2008, Available athttp://www.ttagroup.org.

[10] M. Bofill, M. Palah, J. Suy, and M. Villaret, “SIMPLY: a compiler from aCSP modeling language to the SMT-LIB format,” in Eighth InternationalWorkshop on Constraint Modelling and Reformulation, 2009.

[11] A. Cimatti, A. Franzen, A. Griggio, R. Sebastiani, and C. Stenico,“Satisfiability modulo the theory of costs: Foundations and applications,”in TACAS, 2010, pp. 99–113.

[12] J. Legriel, C. L. Guernic, S. Cotton, and O. Maler, “Approximating thepareto front of multi-criteria optimization problems,” in TACAS, 2010,pp. 69–83.

[13] B. Dutertre and L. de Moura, “The Yices SMT solver,” Tool paper athttp://yices.csl.sri.com/tool-paper.pdf, SRI International, August 2006.

[14] Sha, T. Abdelzaher, K. E. Arzen, A. Cervin, T. Baker, A. Burns,G. Buttazzo, M. Caccamo, J. Lehoczky, and A. Mok, “Real timescheduling theory: A historical perspective,” Real-Time Systems, vol. 28,no. 2/3, pp. 101–155, November 2004.

[15] W. Wang, A. K. Mok, and G. Fohler, “Pre-scheduling,” Real-Time Syst.,vol. 30, no. 1-2, pp. 83–103, 2005.

[16] P. Pop, P. Eles, and Z. Peng, “Schedulability-driven communicationsynthesis for time triggered embedded systems,” Real-Time Syst., vol. 26,no. 3, pp. 297–325, 2004.

[17] A. Metzner, M. Franzle, C. Herde, and I. Stierand, “Scheduling dis-tributed real-time systems by satisfiability checking,” in RTCSA ’05:Proceedings of the 11th IEEE International Conference on Embeddedand Real-Time Computing Systems and Applications. Washington, DC,USA: IEEE Computer Society, 2005, pp. 409–415.

[18] F. Boniol, P.-E. Hladik, C. Pagetti, F. Aspro, and V. Jegu, “A frameworkfor distributing real-time functions,” in FORMATS ’08: Proceedings ofthe 6th international conference on Formal Modeling and Analysis ofTimed Systems. Berlin, Heidelberg: Springer-Verlag, 2008, pp. 155–169.

[19] S. Voss, M. Sorea, and K. Echtle, “SAL-based symbolic scheduling intime-triggered networks,” in IFM ’09: Proceedings of the 7th Interna-tional Conference on Integrated Formal Methods. Berlin, Heidelberg:Springer-Verlag, 2009, pp. 200–214.

[20] L. de Moura, S. Owre, H. Rueß, J. Rushby, N. Shankar, M. Sorea, andA. Tiwari, “Tool presentation: SAL2,” in Computer-Aided Verification(CAV 2004), S. Verlag, Ed., 2004.

[21] H. Kautz and B. Selman, “Planning as satisfiability,” in Proceedings ofthe 10th European Conference on Artificial Intelligence. Wiley, 1996,pp. 359–363.

[22] A. K. Mok and W. Wang, “Window-constrained real-time periodic taskscheduling,” in RTSS ’01: Proceedings of the 22nd IEEE Real-TimeSystems Symposium. Washington, DC, USA: IEEE Computer Society,2001, p. 15.

10