Achieving Realtime Capabilities in Ethernet Networks

by Edge-Coloring of Communication Conflict-Multigraphs

Frank DopatkaFrank.Dopatka@uni-siegen.de

http://www.bs.informatik.uni-siegen.de/

Institute of Operating Systems and Distributed SystemsUniversity of Siegen, Germany

PDCN 2006: Parallel and Distributed Computing and Networks, 15.02.2006

Ethernet in Automation Technology ?slide 2

• Idea: one network from ERP- & office-area (SAP) to the field-devices (sensors, actuators)

• Ethernet: widely-used, easy to use & low cost;CSMA/CD → non-deterministic behaviour

• Automation Technology: determinism, cyclic communication, min. cycle-time, less payload, min. delay & jitter, decentral periphery

• Aim: merge demands of Automation Technologywith the trend towards Ethernet

• TDMA instead of CSMA/CD:industrial-realtime & asynchronous time-slots

• compatibility vs. speed...

Existing Solutionsslide 3

• Beckhoff Automation• standard Ethernet NICs with special

HW-connectors• data transmitted like shifting register→ no real Ethernet any more

• SIEMENS• in-house realtime 4-port switch ASICs→ in-house; 4-ports; packet delays

• Bernecker+Rainer (B&R)• standard Ethernet hubs & central manager → no concurrent communication;

only RT-members

Aim of our Workslide 4

• develop an approach, independent from existing hardware

• support: switches and/or hubs → distributors, half- & full-duplex transmission,decentral periphery → concurrent communication

• using a top-down strategy: - tree-based infrastructure is given- requirements of the devices (QoS-requests) are given

• previously well-known requirements of the devices are e.g.:- Who sends how many data to whom and when ?- What is maximum delay and jitter ?

Aim of our Workslide 5

• first results for a restricted problem-class:- unicast-addressing → Communication Line (CL): sender → receiver- uniform packet-size (Ethmin: less payload)- transmission at each production cycle: no priorization- neither delay, nor jitter in distributors, devices & cables

our scientific contribution:We designed a 4-step-approachfor this restricted problem-class

by using known algorithms!

Survey of our Approachslide 6

network infrastructure

device requirements

Igenerate

Communication Lines

IIbuild conflict multigraph

for each switch

III greedy

edge-coloringheuristics

IVcalculate the

offline-schedules

schedule for each switch

O(n²)

upload to the automated

facilities

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

I: Generate Communication Linesslide 7

Given networking infrastructure with tree-topology...

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

I: Generate Communication Linesslide 8

...with switches, their ports, and devices.

I: Generate Communication Linesslide 9

Line-topology emulated: 3-port-hubs → minimize packet delays

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

I: Generate Communication Linesslide 10

Given requirement R1: dev12 sends to dev17

R1a R1b

I: Generate Communication Linesslide 11

Choosing 1 root-distributor at first for all requirements...

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

R1a R1b

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

I: Generate Communication Linesslide 12

and acquire uplinks from each device to the root-distributor...

R1a R1b

R1b

R1a

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

I: Generate Communication Linesslide 13

concatenation of 2 uplinks → CL 1

1

1

1

1

1

1

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

I: Generate Communication Linesslide 14

Each CL is stored by using an attribution-technique.The root-node is not needed any more.

1

1

1

1

1

2 2

2

33

1

4 4

4

4

6

5

5

55

7 78

899

5

6

65

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

1

1

1

1

1

2 2

2

33

4 4

4

4

6

5

5

55

7 78

899

5

6

I: Generate Communication Linesslide 15

conflict : at least 2 CL share 1 port → common resourceour aim : schedule the conflicts → temporal separation:TDMA

1

65

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

1

1

1

1

1

2 2

2

33

4 4

4

4

6

5

5

55

7 78

899

5

6

slide 16

Why is each switch independent ? → Tree topology ! → Schedules can be synchronized after generation :-)

1

65

II: Build Conflict Multigraph for each Switch

II: Build Conflict Multigraph for each Switchslide 17

Independency of the switches:2 CL in 1 switch have a conflict, or they never have !

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

1

1

1

1

1

2 2

2

33

4 4

4

4

6

5

5

55

7 78

899

5

6

1

65

II: Build Conflict Multigraph for each Switchslide 18

Independency of the switches:A conflict raises never or in exactly one order in the network !

P1

P1 P2 P3 P4 P5 P6

sw4

P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P1 P2 P3 P4 P5

dev6

dev5

P2 P3

sw2

dev7

P1 P2 P3 P4

sw5

dev8 dev9

dev10

dev19

dev18

dev17

P1 P2 P3

h6P1 P2 P3

h7

dev11

dev14dev13

dev12

dev15 dev16

1

1

1

1

1

2 2

2

33

4 4

4

4

16

5

5

55

7 78

899

5

6

65

ports → nodes of the graph P1

P2

P3

P4

P5

P6

P7

P8

II: Build Conflict Multigraph for each Switchslide 19

P1P1 P2 P3 P4 P5 P6 P7 P8

sw1

dev2

dev3 dev4

dev1

sw3P2

dev6

dev5

sw212 23

34 4

16 7 78

899

5

65

23

4

5

6

7 8

9

1

CLs → edges of the graph

slide 20

1

2

4

3 5

6

7 8

9

P1

P2

P3

P4

P5

P6

P7

P8

WHILE (not all edges colored)x := any non-colored edgeM := set of neighbor-edges from xcolor(x) := lowest color not present in M

END WHILE

III: Greedy-Edge-Coloring Heuristics

Color 1:

Color 2:

Color 3:

IV: Calculate the Scheduleslide 21

1

2

4

3 5

6

7 8

9

P1

P2

P3

P4

P5

P6

P7

P8

parallelism

cycle time

color / time slot1 2 3

137

258

469

First Resultsslide 22

• Time complexity with constant amount of CLs reduces logarithmical with the amount of ports !

→ lower „density“ of the graph→ easier to find next free color for greedy

• We generated 32 random conflict-graphs each with- 4 / 5 / 6 / 8 / 12 / 16 / 24 / 32 ports- 2000 / 4000 / 8000 / 16000 randomized CLs→ greedy-coloring on 2.4Ghz Intel P4-Laptop

• 4000 CLs in a switch can be calculated in 3 sec.: O(n2)

First Resultsslide 23

Problems with oddamount of nodes/ports & circles in graph:

12% more colors than theoretical lower bound on average...

otherwise 0.1% more colorson average

But: - cyclic transmission unusual in Automation Technology- 5-port-switch: only 2 CLs at the same time anyway!

Conclusionslide 24

• We found a method for a restricted problem-class: - unicast-addressing- uniform packet-size- sending at each cycle- neither delay, nor jitter

• schedules nearby theoretically best-case resultsif no odd amount of ports are used

• non-realtime traffic options can be added by additional time-slots

Conclusion & Future Workslide 25

• include more common cases → relax restrictions- multicasting & multiple packet sizes- sending at each 2nd, 3rd,... cycle- handle delay & jitter

• improve the coloring by usage of known additional heuristics→ only necessary for distributers with most colors

• consider, where the schedule is enforced - inside the switches & dev-polling („Laptops allowed“)- using external arbiters & dev-polling („standard hardware“)- enforcing inside the devices („fast“)→ consequences for the generation of the schedules

• simulate industrial case scenarios with own framework

Thank you for your Attention!

Any Questions?