Control in Printing Systems: Modular Reconfigurable Media Paths

Haitham Hindi and Lara S. Crawford

Palo Alto Research Center (PARC)

Palo Alto, California, USA

{hhindi,lcrawford}@parc.com

Abstract— The remarkable drop in the cost of embeddedcomputing, sensing, and actuation is creating a correspondingexplosion in the use of these technologies in new systems. Inorder to reduce complexity and contain costs, manufacturersare moving toward modularizing their systems and buildingreconfigurable products from simpler but smarter components.Of particular interest have recently been highly reconfigurablesystems, i.e., systems that can be customized, repaired, andupgraded at a fine level of granularity throughout their lifetime.

High reconfigurability puts new demands on the dynamiccalibration, control, and coordination software in the system.There is much promise in existing software approaches, butcurrent techniques face a number of new challenges beforethey can be embedded in the kind of real-time, distributed, anddynamic environment found in highly reconfigurable systems.In this tutorial paper, we will discuss challenges, solutions, andlessons learned in the context of a long-term project at PARCto bring such techniques to a highly reconfigurable paper pathsystem.

I. INTRODUCTION

This paper presents several aspects of the control of a

high-speed, highly modular distributed printing system. Such

a system presents several challenges from a control point of

view. The high-speed sheet transport and distributed embed-

ded nature of the controller necessitate control algorithms

that are efficient in terms of tracking performance, processing

time, and communication overhead. The highly distributed

nature of the system also introduces a coordination challenge,

as several actuators are acting on a sheet of paper at any given

time. This coordination must be achieved under conditions

of finite bandwidth and network delays.

First we will describe the physical system and control

architecture for the tightly coupled, distributed system. We

will then present an overview of the control and coordination

algorithms developed for this system. Next we will describe

the software architecture and the interactions among the

software components. Finally, we will show some results

from the real system. This paper draws heavily from [16],

[10], but also conttains previously unpublished material.

II. HIGH-SPEED PARALLEL PRINTING TESTBED

The highly modular printing system is shown in Figs. 1

and 2. It contains four black and white 55-page-per-minute

print engines that do the printing, two paper feeders, and

three outputs, all linked together with custom-designed paper

path modules. There are three high-speed sheet “highways”

in the system, which run sheets at 1.25 m/s, with “off-ramps”

and “on-ramps” sending paper to and from the marking

Fig. 1. The hypermodular high-speed printer prototype.

engines (which run at a fixed slower speed, 0.257 m/s). Many

cross-connections link the highways. At full capacity, the

printing system can produce 220 sheets per minute and can

have up to 15 sheets being processed simultaneously.

There is an independently controlled roller actuator (called

a “nip”) module between each section of bidirectional paper

path. The nips are driven with stepper motors. There is one

infrared sheet-edge-detection sensor on either side of each

nip (not shown in the diagram). Each director element (three-

way paper path module) also has its own controller, to control

the solenoid flipper actuators that direct the paper on different

paths. Each of the “straight-through” paper path modules also

has a processor, though it is not currently used for control

purposes. Finally, the feeders and print engines each have a

processor as well. There are over 150 independent processors

in the system, along with 90 nip motors and 180 sensors.

The modules communicate via multiple CAN buses. Multiple

buses were required for reasons of loading, and also proved

useful for bandwidth reasons. The prototype is designed to

be highly modular and reconfigurable, in both hardware and

software.

III. CONTROL ARCHITECTURE

In our printing system, the low-level control (module level)

is distributed for reasons of modularity and flexibility, but

maybe just as importantly, sample rate and bandwidth/sensor

2009 American Control ConferenceHyatt Regency Riverfront, St. Louis, MO, USAJune 10-12, 2009

WeA02.1

978-1-4244-4524-0/09/$25.00 ©2009 AACC 38

Fig. 2. Diagram of the hypermodular printer prototype. Green lines indicatepaper path. Gray boxes indicate paper path module boundaries. Blue doublecircles represent nip rollers. The thick red line shows a typical sheet path.

rate considerations. A tight level of coordination is required

among these distributed actuators, however. As a sheet pro-

gresses through the machine, a different group of actuators

will be in contact with it at different points in time. These

actuators form a dynamic control group, and must all act

precisely in synchrony to avoid damage to the sheet and

potential paper jams. For example, a nominal sheet speed

might be 1 m/s. To support a variety of sheet sizes, the

spacing between nips might be on the order of 0.1 m. An

8.5 x 11 inch sheet of paper moving through the system

long edge first would be in contact with an individual nip

for approximately 0.2 s. Given two nips spaced 0.1 m apart,

it would be in contact with those two nips simultaneously for

0.2−0.1 = 0.1m of travel, or 0.1 s. In that amount of time,

a 10% velocity discrepancy between the two nips translates

into 1 cm of position differential, assuming no slippage. If

the trailing nip were the one rotating faster, this differential

would probably result in buckling the sheet, and possibly a

paper jam. If the lead nip is moving faster, the sheet risks

surface damage or simply poor control tracking.

Since the sensory feedback is event-based and sporadic

in this system, a significant potential source of velocity dis-

crepancy among nips is different controllers applying sensory

feedback at different times. If, for example, a sensor signal

were obtained at a local controller, acted on immediately, and

also passed to other controllers in the control group, those

controllers would act on the information with some delay.

The delay depends on the networking architecture and the

update rate of the controllers. During the delay, a control

deviation will only produce a small position discrepancy,

but such errors corrupt the synchronization of the controller

states, and the discrepancy can grow over time. In our sys-

tem, there are two sensors for every nip, so sensor crossings

are quite frequent. It is therefore important to ensure that the

actuators do not apply sensory feedback at different times,

and do act in complete synchrony.

These tight coordination requirements between the mod-

ules dictate the existence of some kind of coordination

entity. This entity, the sheet controller, serves as a kind of

proxy for a given sheet from the point of view of higher

u1 u2 u3 u4

MC−2 MC−3 MC−4MC−1

x1 x2 x3 x4

Mod−1 Mod−2 Mod−3 Mod−4

Planner

Sheet

Controller

sheet

<< local area network>>

<<controller area network>>

u

module1

module2

module3

t

t

module2

module3

module1

x

<module1, operation2, 3.2 seconds>

<module2, operation1, 3.7 seconds>

<module3, operation1, 4.2 seconds>

Fig. 3. Hierarchical networked control system. The thin lines showcommunication pathways. A specific example is shown in thicker lines:a sensor (small square) generates information that is passed up to the sheetcontroller, which then relays it to all modules that need the feedback. Theright part of the figure shows plan translation and distribution by the sheetcontroller. The planner sends a plan in the form of a list of tuples to the sheetcontroller, which converts it into trajectories for the modules to track. Duringtimes of overlap, the trajectories are identical. The module controllers trackthe identical trajectories, producing synchronized actuator outputs.

levels of control, and is responsible for the sheet’s safe

passage through the machine. The sheet controller monitors

the progress of the sheet and serves as an information

broker between the module controllers by relaying sensor

information to appropriate module controllers via a publish-

subscribe protocol over the network.

Based on these considerations, we arrived at the hier-

archical control architecture shown in Fig. 3. The plan-

ner/scheduler specifies a series of way points which are

desired arrival times and locations for the leading edge of

the sheet of paper. This information is then passed (over

a network) to a sheet controller which processes it and

passes trajectory commands to the relevant module con-

trollers (again over a network). There is one sheet controller

per sheet – a new controller is created when instructions

for a new sheet are received from the planner. Since the

exact arrival time of the sheet is not known in advance

(due to noise disturbances), the sensing is asynchronous. The

module controllers feed the sensor information to a control

algorithm (PTOS algorithm [11]) and then apply the resulting

control signals to the rollers (nips), which move the paper.

The module controller algorithm is shown in Fig.4, and will

be described in more detail later. The nips are actuated by

stepper motors with extremely fast dynamics compared to

the sheet system. The paper is in contact with more than

one nip at any given time. This example system is discussed

in more detail in Fromherz, et al. [12]. Further control issues

in printing systems can be found in Hamby and Gross [14],

Li, et al. [20], Chen and Chiu [7], and Krucinski, et al. [18].

The control goals of the system are to deliver tight control,

which appears to the paper as good as centralized control,

in the presence of random network delays and limited

bandwidth. The desired behavior can be seen in Figs. 5 and

6. Fig. 5 shows an example of a nominal sheet trajectory

and the times the different nips are active. The trajectory is

39

Fig. 4. Block diagram of the module controller algorithm. A time varyingKalman filter handles the irregular sensing rate; proximate time optimalcontroller PTOS handles the actuator saturation; delay equalizer handles thebounded network delay; and synchronization statemachine enables controllerhandoff for distributed implementation.

0 0.5 1 1.5 2 2.5 30

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Nip−0

Nip−1

Nip−2

Nip−3

time (s)

positio

n (

m)

Fig. 5. An example of a nominal sheet trajectory obtained by directinterpolation of way points (asterisks). Leading and trailing edge trajectoriesare both shown. The dark dotted lines are the module boundaries. Thesquares on the y-axis are the positions of the edge sensors. The x-es andcircles on the x-axis show the nominal leading edge and trailing edge,respectively, sensor crossing times. Also shown are the nominal states ofthe FSM: dotted=Synch, dashed=Comp, solid=Drive.

given by the way points, computed by the sheet controller

and shown as asterisks, and marks the time the leading edge

of the sheet should nominally arrive at a point. The time the

trailing edge of the sheet should leave a point is also shown.

The dark bands connecting these two trajectories show the

times that different nips should be active. The sheet controller

uses these trajectories to compute the activation times for the

nips. The desired control behavior for the system can be seen

in the simulation results shown in Fig. 6. The active control

signals for four different actuators are shown as solid lines

on the four plots; the dashed trace is the same on all plots,

and is the control output that would be used in a centralized

system (the desired control output from all the distributed

controllers). The actuators activate and then deactivate in

sequence. The states of the controllers, also shown in Figs. 5

and 6, will be discussed in Section VI.

One of the key design decisions in our networked system

architecture was the practical question of how to implement

the control algorithm mentioned above in this distributed

0 0.5 1 1.5 2 2.5 3

00.10.20.3

Nip

−0

time (sec)

0 0.5 1 1.5 2 2.5 3

00.10.20.3

Nip

−1

0 0.5 1 1.5 2 2.5 3

00.10.20.3

Nip

−2

0 0.5 1 1.5 2 2.5 3

00.10.20.3

Distributed Nip Control

Nip

−3

Fig. 6. Overall nip speed control signal (dashed) along with the drivesignals (solid) superimposed for the first four nips. Along the x-axis, theline styles show the states of the FSMs (dotted=Synch, dashed=Comp,solid=Drive).

actuation and sensing environment. There are at least two

alternatives. One reasonable method might be to implement

the controller centrally in the sheet controller, and simply

communicate the control signals to the appropriate nips via

the module controllers. However, in our application, the

control updates happen much more frequently than sensor

messages, so a lot of network bandwidth could be saved by

transmitting only the sensor messages.

We have instead, therefore, chosen a more distributed

implementation, at the cost of redundancy. In this distributed

implementation, each module controller runs its own local

replica of the control algorithm, i.e., the same controller

is used in all the nips in contact with the sheet. When

properly initialized, if all these controllers receive the same

sensor information, they will apply the same control signal

to the sheet. As the sheet moves through the system, new

nips will constantly be joining and leaving the actuation

of the paper. Hence there is a need for a synchronization

method, which can be viewed as a mechanism for controller

hand-off. As we shall show, this synchronization can be

effected by embedding the control algorithm in a finite state

machine (FSM), which takes care of the synchronization (see

Fig. 14 ). Thus the sheet moves as if being controlled by a

single centralized controller running the algorithm in Fig.4.

Consequently, the sheet controller only needs to send very

infrequent messages to the module controllers consisting of

high level commands telling the nips when to switch on and

off and from handling one sheet to another and relays of the

relatively infrequent sensor information. Network bandwidth

is thus used more efficiently.

IV. DYNAMICAL CONTROLLER

In this section, we show how classical closed form time

optimal control [19], [6], [2], [11], which exploits the special

40

structure of double integrator systems (i.e. ones with point-

mass Newton’s Law dynamics) can be used to design a

hybrid controller for waypoint tracking in the highly modular

printing system, for example. While double integrator dy-

namics are very simple, they are extremely prevalent in many

domains, beyond manufacturing systems, e.g., transportation,

disk drives, robotics, and aerospace. Our approach is based

on appropriate setting (switching) of the state of a reference

generator with the same point mass dynamics in a two degree

of freedom controller topology [19], [6], [2], [28], [11]. It

can be viewed as a special case of the more sophisticated

modern methods available today, see for example [4], [8],

[3], [5]. However, the techniques we present admit very

compact implementations, suitable for use in low cost micro

controllers and DSP chips used in modular reconfigurable

embedded systems applications.

The goal of the control is to drive paper along the paper

paths shown, while meeting waypoint tracking requirements.

The waypoints for each sheet of paper, along with the routing

and scheduling, are determined by the planner/scheduler, as

discussed above.

A. PROBLEM STATEMENT

We will be concerned mainly with the double integrator

system:s = v

v = u(1)

where s and v are the position (m) and speed (m/s), respec-

tively; u is the acceleration control signal (m/s/s). If we define

the plant state xP = (s,v), then (1) can be written in state

space form as

xP(t) = AxP(t)+Bu(t), xP(t0) = xP0

sP(t) = Cxp(t)(2)

where

A =

[

0 1

0 0

]

, B =

[

0

1

]

, C =[

0 1]

.

Note that [A,B] is controllable, and the system has an

invertible time evolution matrix

Φ(t0, t f ) = eA(t f −t0) =

[

1 (t f − t0)0 1

]

.

In many modern control systems, such as the one in Fig. 2,

an abstract planner handles task ordering and scheduling, and

then passes a set of waypoints to the physical controller.

These are a set of specific times, locations, and speeds,

{(t0,s0,v0), . . . ,(tN ,sN ,vN)} for the system to achieve at

certain times. Suppose we view the way points as sam-

ples at instants {t0, . . . , tN} of a fictitious reference state

trajectory xR(t), whose values at those times are xR(t0) =(s0,v0), . . . ,xR(tN) = (sN ,vN). Then we may state the way-

point tracking problem in its most skeletal form as

minu

∑ti∈{t0,...,tN}

||xP(ti)− xR(ti)||p

s.t. xP(t) = AxP(t)+Bu(t); xP(t0) = (s(t0),v(t0))||u||∞ ≤ amax

module

amax

(t0, s0, v0) (tf, sf, vf)

sf−s0

Fig. 7. Single module waypoint tracking

This captures our primary objective, namely that of hitting

the waypoints without exceeding our motor peak acceleration

constraints.

As systems become more modular, and as the modules

themselves become more autonomous, eventually, one arrives

at the following single module waypoint tracking problem,

which will be the focus of this paper. The multiple waypoint

tracking problem above can be decomposed into a sequence

of single module problems, possibly by using tricks such as

fictitious modules, to handle nonuniform spacing, etc.

Problem Statement: [single module waypoint tracking]

Given a module with double integrator dynamics (2), find a

control u : R2 → R that will take the state from xP0 = (s0,v0)at t0 to xP f = (s f ,v f ) at t f , while not exceeding the peak

acceleration limit amax. If such a control u can be found

from a certain class of control laws, then we say that the

pair {(t0,xP0),(t f ,xP f )} is amax-feasible under that class of

control laws. See Fig. 7.

Note that the above only captures the most critical con-

straints. In practice, one would like to have all the following:

1) pass through the waypoints (or as close as possible),

2) don’t exceed acceleration limits

3) easy to check that {(t0,xP0),(t f ,xP f )} is amax-feasible

4) feedback implementation to reject disturbances

5) low memory and computation costs for DSP imple-

mentation

6) compact parametrization to fit network bandwidth lim-

its

7) minimal transit time through overall path

8) smooth machine-friendly position and velocity trajec-

tories

9) speed should always be positive (no moving back-

wards)

B. 2DOF CONTROL TOPOLOGY

We now introduce the basic idea that we use to solve

the waypoint tracking problem. Specifically, we will make

use of the two degree of freedom controller (2DOF) topol-

ogy, Fig. 8, which uses a reference generator to provide

feedforward control, and a state feedback for stabilization

and disturbance rejection. This is a classical architecture,

discussed in many standard references [19], [6], [2], [28],

[11]. The equations for the overall system are now

xR = AxR, xR(t0) = xR0

xP = AxP +Bu, xP(t0) = xP0

e = Cxp −CxR,

u = f (xP − xR).

(3)

In this section, we assume all states directly available and

we focus on the designing a full-state feedback/feedforward

41

xR0 R

Pf (·) ux

xP

xR

sP

sR

e

Fig. 8. Two degree of freedom controller topology. The controller consistsof R and f (·); takes 2 inputs: initial condition xR0 and plant state xP; andproduces the control output u = f (x).

control law; observer implementation is discussed in the next

section.

In the hypermodular printer fixture in Fig.2, each stepper

motor is controlled independently by its own DSP processor,

which runs such a 2DOF controller.

Proposition 1: Consider the problem of tracking the state

of a linear reference generator, as in (3), whose state is fully

observable, with the same dynamics A-matrix as the plant.

Let u : Rn → R be a stabilizing (possibly nonlinear) state

feedback for the plant. Let x := xP − xR. Then the control

u(x) will stabilize the tracking error x and drive it to zero

with the same closed loop dynamics as it has on the plant

by itself.

Proof : Trivially we observe that subtracting the two state

equations in (3) gives

˙x = Ax+Bu(x)

which has exactly the same dynamics as the original system

in closed loop with the stabilizing state feedback u.�

Proposition 2: Assume that Φ(t0, t f ) = eA(t f −t0) is invert-

ible and that [A,B] is controllable. Set xR(t0) = eA(t0−t f )xP f .

Then if u is a control law which takes the state x := xP −xR

to zero in finite time t∗ ∈ [t0, t f ], then u(x) solves the single

module waypoint trajectory control problem.

Proof : Since Φ is invertible, we can propagate xR back

in time from t f to t0. Now from time t∗ onwards, we

have xP(t∗) ≡ xR(t∗). By controllability, this means that u ≡0,∀t ≥ t∗. Thus xP and xR will both evolve homogeneously

according to xP(t) = xR(t) = eA(t−t∗)xR(t∗),∀t ≥ t∗. Since the

reference generator was evolving homogeneously all along,

we have xR(t∗) = eA(t∗−t0)xR(t0) = eA(t∗−t0)eA(t0−t f )xP f =eA(t∗−t f )xP f . Substituting into the expression for xP(t) and

evaluating at t = t f gives xP f . �

Thus given any finite time stabilizing control law u for the

double integrator system, we have a paradigm for solving the

single module waypoint tracking problem as follows:

1) check points are amax-feasible

2) set xR(t0) = eA(t0−t f )xP f

3) apply control law u(x) from t0 to t f

C. TIME OPTIMAL CONTROL BASED TRACKING

We will now review some classical results on time optimal

control for double integrator systems, which will supply

us with a finite time stabilizing control law, that can be

used in the propositions above to design waypoint tracking

controllers, see [19], [6], [2], [11].

The time optimal control problem is

minu

T =∫ t0+T

t0

1 dt

s.t. x(t) = Ax(t)+Bu(t); x(0) = xo

x(T ) = 0

‖u‖∞ ≤ amax

The solution to the time optimal control problem for the dou-

ble integrator system is a nonlinear state-dependent control

law

u(t) = fTOC(x(t)) (4)

= −amaxsgn(

sgn(s(t))√

2amax|s(t)|+ v(t))

where recall that the state x(t) = (s(t),v(t)).At this point one might be tempted to apply u(t) =

fTOC(x(t)) right away to the plant and reference system to

solve the problem. But first we need to be able to check

amax-feasibility of a given pair of waypoints under the control

law fTOC(x(t)). Fortunately, once again thanks to the special

structure of the double integrator system, there is a closed

form expression for the time optimal transfer time: For the

double integrator system (2), the time taken to bring the state

from x0 to the origin under the time optimal control law (4)

is given by

tTOC(x(t0))=

v(t0)+2

√

s(t0)+v(t0)2

2amax;s(t0) > −

v(t0)|v(t0)|2amax

−v(t0)+2

√

−s(t0)+v(t0)2

2amax;s(t0) < −

v(t0)|v(t0)|2amax

|v(t0)| ;s(t0) = −v(t0)|v(t0)|

2amax

(5)

Using these facts together with Prop1 and Prop2, we

can propose the following controller for the single module

waypoint tracking problem:

Proposition 3: Consider the double integrator system,

with acceleration limit amax, together with the reference

generator system as in (3). Let x := xP − xR, and suppose

we want to move from xP0 = (s0,v0) at t0 to xP f = (s f ,v f )at t f . Then by taking xR(t0) = eA(t0−t f )xP f , the control

u(x) = fTOC(x) (6)

will take the state from (t0,xP0) to (t f ,xP f ) while not

exceeding the peak acceleration limit amax, if and only if

tTOC(x(t0)) ≤ t f − t0. (7)

where x(t0) = xP(t0)− xR(t0).Remark: Note that here evaluating xR(t0) = eA(t0−t f )xP f is

merely setting

xR(t0) =

[

1 (t0 − t f )0 1

][

s f

v f

]

=

[

s f − v f (t f − t0)v f

]

.

In other words, we simply extrapolate back the desired

output trajectory, start the reference generator on that desired

42

trajectory, and then apply the control that will move the

system from its input trajectory to the desired one in minimal

time.

Unfortunately, this control approach results in excessive

switching which can be harsh on the mechanics, and can also

produce trajectories where the sheet moves in reverse, for

short periods of time, thus violating some of the secondary

design considerations above. The method we describe next

reduces the switching and eliminates the backwards motion.

�

D. INFLECTION FREE CONTROL BASED TRACKING

In this section we will describe another waypoint tracking

control method which has smoother profiles, at the expense

of some restriction on the feasible set of state transfers.

Similar ideas can be found in standard references, eg: [9].

However, we have not seen elsewhere the closed loop im-

plementation via time optimal control, presented here. The

resulting controller contains switching not only due to the

time optimal control law, but also due to deliberate switching

of the state of the reference generator. It can be viewed as

a practical example of a simple hybrid systems switching

controller.

Specifically, in some applications, it is known ahead of

time, or by design, that the overall function of a certain mod-

ule is either to slow down or to speed up. In this case, one

can design inflection free trajectories. These are trajectories

where the acceleration does not flip sign and hence are either

convex or concave. We will design controllers for a specific

subclass of inflection free trajectories.

Assumption: For the rest of this section, we will assume

that it is required to keep the controlled objects moving in

the forward direction, ie, we will not allow the velocity to

reverse.

Definition 1: A constant acceleration inflection free

(CAIF) trajectory is one which is generated by applying

a constant acceleration amax for some subinterval of time

[t ′0, t′f ] ⊆ [t0, t f ], see Fig.9.

By integrating the area under the curves, one can see that

the trajectory will have the following form:

s(t)=

s0 + v0(t − t0) ; t0 ≤ t < t ′0s0 + v0(t − t0)+ 1

2 amax(t − t ′0)2 ; t ′0 ≤ t < t ′f

s0 + v0(t′f − t0)+ 1

2 amax(t′f − t ′0)

2 + v f (t − t ′f ) ; t ′f ≤ t < t f

(8)

The next question is: what pairs of waypoints are CAIF

amax-feasible? Since (8) gives us the explicit expression for

CAIF amax-feasible points, then we can check the feasibility

of a given waypoint pair ((t0,s0,v0),(t f ,s f ,v f )) by checking

if they are consistent with (8). Specifically, if we evaluate

(8) at t = t f , we obtain the expression

s f = s0 + v0(t′f − t0)+

1

2amax(t

′f − t ′0)

2 + v f (t f − t ′f ).

Using this together with the observation (see Fig.9)

amax =v f − v0

t ′f − t ′0⇒ t ′f = t ′0 +

v f − v0

amax(9)

t t ’ t ’ t

v

v

s

s

0 0 f f

o

f

o

f

a

v

v

o

f

Fig. 9. Constant acceleration inflection free trajectory.

t0 t ’0 tf

t ’f tf

t ’f

t0 t ’0

v

v

0

f

v

v

0

f

=

=

amax

amax

Fig. 10. Extreme cases: given a fixed amax, the top figure shows the minimalpossible change in distance (smallest possible area under the velocity curvefor amax); the bottom figure shows the maximal possible distance change(maximum possible area).

we can solve for the switching times as functions of((t0,s0,v0),(t f ,s f ,v f )):

t ′0 =

(

1

v f − v0

){

v f t f − v0 t0 − (s f − s0)−1

2amax(v f − v0)

2

}

(10)

t ′f =

(

1

v f − v0

){

v f t f − v0 t0 − (s f − s0)+1

2amax(v f − v0)

2

}

.

Therefore, the waypoint pair ((t0,s0,v0),(t f ,s f ,v f )) being

feasible and consistent with (8), is equivalent to plugging it

into (10) and obtaining valid switching times which satisfy:

t0 ≤ t ′0 and t ′f ≤ t f . (11)

Equivalent consistency conditions can be obtained by in-

specting the extreme cases in Fig.10. Noting that the distance

covered is the area under those curves, one obtains the

following conditions, which turn out to be exactly equivalent

to (10) and (11):

v0∆t +δ ≤ ∆s ≤ v f ∆t −δ ;v0 ≤ v f

v f ∆t +δ ≤ ∆s ≤ v0∆t −δ ;v0 > v f(12)

43

0 0.5 1 1.5 20

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

v0 (m/s)

vf

(m/s

)

(v0, vf) Pairs Achievable with aMax = 10 m/s/s, dx = 0.1 m

dt=0.06

dt=0.08

dt=0.14

Fig. 11. As we sweep ∆t from 0 to 0.15, for each value, (12) producesa pair of “lemon” shaped regions, corresponding to the set of (v0,v f ) pairsthat are amax CAIF feasible for that choice of ∆t and the fixed ∆s and amax.Also shown for comparison in the cyan dotted lines are the regions of thecontrol law in the previous section.

where ∆v = v f −v0, ∆t = t f − t0, ∆s = s f −s0, and δ = ∆v2

2amax.

Fig.11 shows the feasible input and output speeds for a

module of fixed length and a given amax, computed using

(12), which is easy to check in a low cost processor. Pairs

that are outside the boundaries are infeasible. Furthermore,

if we ignore the waypoint timing constraint and simply ask:

what pairs ((s0,v0),(s f ,v f )) are CAIF amax-feasible for some

∆t = t f − t0, one obtains the condition:

1

2amax(v2

f − v20) ≤ (x f − x0)

which is shown as the solid boundary lines in Fig.11. This

follows from (9) and Fig.10 & Fig.9, which show that

any CAIF curve must contain at least the trapezoidal area

(distance) between t ′0 and t ′f , which is 12amax

(v2f − v2

0). Also

(9) gives us the minumum ∆t required to achieve ∆v using

amax.

Proposition 4: Given amax, the waypoint pair

((t0,s0,v0),(t f ,s f ,v f )) is CAIF amax-feasible if and

only if (11) or (12) hold. Then the double integrator system

can be made to track the CAIF trajectory using the 2DOF

architecture with the time optimal control law by setting

xR(t0) = xP(t0) = (s0,v0) and applying the control law

u(x) = fTOC(x); then at time t = t ′0 given by (10), the

reference generator state is switched to track the desired

output trajectory, xR(t ′0) = eA(t ′0−t f )x f = (s f −v f (t f − t ′0),v f ),while continuing to apply u(x) = fTOC(x) all along.

Proof : By construction, the amax CAIF curve is the result of

applying amax to the double integrator on the interval [t ′0, t′f ].

Since the TOC control will attempt to drive x to zero in

minimum time, it must also apply amax for that same duration

- it cannot do anything else to reduce the error any faster. �

Thus if we use (6) for f (·) in the 2DOF architecture of

Fig.8, with the reference generator switching as described

above, the resulting controller now satisfies all of our desired

practical criteria above, see Fig.12. Of course the satisfaction

of all of the constraints comes at a cost: the inflection free

property of the trajectories reduces the set feasible end-

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18

−0.1

−0.05

0

0.05

0.1

sR

,sP

t

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18

0

0.5

1

v

t

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18−20

−10

0

10

20

a=

u

t

Fig. 12. Tracking using the CAIF control of the waypoints((0.0,0.0,0.25),(0.18,0.10,1.0)). (Top) position plant (red) and reference(blue). (Mid) plant speed. (Bot) acceleration control.

to-end transfers, see Fig.11; the reference switching adds

(minimal) complexity to the controller; and there is a small

additional messaging bandwidth cost: once feasibility has

been established, instead of sending x at t0, we must now

send xR(t ′0) and also include t ′0. Nevertheless, we have found

these tradeoffs quite acceptable, in order to achieve the specs

of smoother trajectories and no backwards motion.

Remark: Note that in the absence of noise, the optimal

control will be zero on the initial interval [t0, t′0] since both the

reference and the plant have the same dynamics and initial

conditions, and hence the tracking state error x will be zero.

The control will be nonzero on the interval [t ′0, t′f ], during

which the tracking error will be reduced to zero. Then again

after t ′f , the control will again be zero. See Fig.12. Of course,

the benefit of having the closed loop control implementation

is that, in practice, the system will always be subject to noise

and disturbances, and thus there is the opportunity for the

control to correct for these perturbations.

Remark: Our presentation has been in continuous time;

however, in practice, we have implemented our systems in

discrete time. There are various subtleties in approximating

continuous time optimal control with discrete time-optimal

control. However there are solutions too. We refer the reader

to [11] for a discussion of these issues. We have found the

proximate time optimal servo (PTOS) described there to be

quite satisfactory in addressing the main issues in discrete

time implementation.

Remark: Note that we have derived the conditions for

switching in terms of times t ′0 and t ′f . One could just as

well compute corresponding transition conditions in terms of

44

Fig. 13. Delay equalizer removing variations in round-trip message delay,as applied to sensor messages. Messages to the control group are routedthrough the sheet controller. The delay equalizer at each module controllerholds a received message until an apply time ta specified by the sheetcontroller. ta is the sensor trip time plus a maximum round trip networkand processing delay equal to d controller time steps.

the state xP. It would be interesting to compare the hybrid

robustness properties of both approaches, along the lines of

[25].

V. DISTRIBUTED NETWORKED

IMPLEMENTATION

As described above, these module controllers commu-

nicate to the sheet controllers over a network, and must

therefore be synchronized in the presence of network delay.

We assume that there is a worst case network and processing

delay describing the time it takes for sensor or controller state

information to be passed up from the module controller to

the sheet controller and back down over the multiple parallel

CAN buses. Putting the sheet controller in the loop increases

the delay, but enables the modules to be ignorant of the

machine configuration. The delays can then be made constant

for each message type through the use of a delay equalizer

mechanism [12], [23], [22], [21]; see Fig. 13. In the modular

printing system, the state information has an equalized delay

of D control samples, D ∗ Ts, and the sensor data has an

equalized delay of d ∗ Ts, where d ≤ D due to the relative

message sizes. These delays are significant in terms of the

control sample time, and affect controller synchronization as

well as the controller design. The module controller’s delay

equalizer simply accepts new incoming messages and delays

them further until the apply time, ta, when the total delay

since their origination time ts is d or D time steps.

The synchronization problem we are solving is to deter-

mine what information to send over the network and when

to send it in order to initiate and maintain synchronization,

given the network delays. Ideally, we want to send the

minimum amount of information that is sufficient for syn-

chronization. The synchronization protocol is quite general

and does not depend on the specific tasks of the controllers.

Consider, then, a set of processes all following a recursion:

x(k +1) = f (x(k),y(k−d),k) ; x(0) = x0

u(k) = g(x(k),y(k−d),k)(13)

for any functions f and g of x, y and k. We refer to such

a set of processes, where the x(k) and u(k) are identical

for all time, as synchronized. For the control group to be

synchronized, it is sufficient that each controller:

• has a synchronized clock,

• runs an identical algorithm, as above,

• is properly initialized when joining the control group,

so that its state matches the state of the control group,

• receives identical commands, and

• receives identical sensory information.

Clock synchronization is assumed here, and is discussed in

[10] as well as voluminously elsewhere (see, for example,

[13], [27]). The sheet controller ensures that the processes

(module controllers) receive identical commands. It also

ensures that the control group receives the same sensory

information, and uses it at the same time, using a delay

equalizer (Fig. 13) as described above. This section describes

the process of synchronization initialization.

In the following we apply the general synchronization

technique from [17] to an observer-based controller. Consider

a controller with an observer-based structure, which has a

state of the form x(k1|I (k2)), the best-estimate state at

time k1 given information available up to and including time

k2. As a shorthand, we will write x(k1|I (k2)) as simply

x(k1|k2). Note that x(k1|k2) will include sensor information

generated only up to time k2 − d, due to the sensor delay.

Assume that the estimation update operation, performed

when new data y(k− d) is received at time k, acts on the

state estimate from d time steps ago:

x(k−d|k) = fu(x(k−d|k−1),y(k−d),k) (14)

This update can be a Kalman filter or some other type of

update function. In the printing system the sensors were ac-

curate enough that using the sensor position data directly was

sufficient, while a Kalman filter added a lot of computational

overhead for not much improvement. After the estimation

update, in order to bring the improved state estimate forward

to the present time the controller must propagate x(k−d|k)forward using the one-step time update function ftu:

x(k−d +1|k) = ftu(x(k−d|k),u(k−d),k−d)

... (15)

x(k|k) = ftu(x(k−1|k),u(k−1),k−1)

These operations, fu followed by d iterations of ftu, together

comprise the measurement update, fmu:

x(k|k) = fmu(x(k−d|k−1),y(k−d), (16)

u(k−d), . . . ,u(k−1),k)

Note that if no measurement y(k − d) arrives in a given

time step k (as the measurements are asynchronous), the

measurement update step can be skipped. Next the controller

must calculate u. Assume its calculation is of the form:

u(k) = gu(x(k|k),k) (17)

Finally, the controller needs to perform one further time

update to obtain the new current state:

x(k +1|k) = ftu(x(k|k),u(k),k) (18)

In the hypermodular printing system, we implemented these

calculations using history buffers for x and u.

45

Now we will consider how to synchronize controllers of

this type.

Proposition 5 (Augmented State): Given a con-

troller of the observer-based form described above,

representing its state in the augmented form:

x(k) = (x(k−d|k−1),u(k−d), . . . ,u(k−1))

yields state updates of the form of the recursion in

(13).

The proof follows from analyzing the dependencies of the

update functions in (14) – (15), and is given in more detail

in [10].

Now, given the sensor message delay of d steps and

the state message delay of D steps, we have the following

conditions required for synchronization [17]:

Proposition 6 (Synchronization): Let

{p0, . . . , pn−1} be a set of processes running

(13). A new process pn, which knows f and

g, can be synchronized with the given set

from time k′ onwards, and for any functions

f (x(k),y(k− d),k) and g(x(k),y(k− d),k), if and

only if the following conditions hold:

A. pn receives the same input measurements

{y(k−d) | k ≥ k′} and

B. k′ ≥ D and, at time k′, pn has access to

x(k′ − D) and {y(k′ − D − d), . . . ,y(k′ − 1 −d)}, for synchronization via forward propa-

gation. Forward propagation consists of per-

forming D iterations of the state recursion

(13) to obtain x(k′) from x(k′ −D) and the

input measurements {y(k′−D−d), . . . ,y(k′−1−d)}.

The proof follows from performing the forward propaga-

tion, and is given in [17].

In the modular printing system, in order to ensure property

A in Proposition 6, that the controllers receive identical

sensory input, all sensory information is routed through the

sheet controller and equalized to d time steps of delay, as

described above (Fig. 13). The sensor input is based on sheet

edge crossings, and is thus asynchronous (sensor readings do

not arrive at every time step).

Property B in Proposition 6 addresses the means for

initializing synchronization for a new controller joining a

control group. In the printing system, it is not sufficient to

synchronize all controllers at the beginning of operation, or

even perform synchronization updates at regular intervals,

as there are multiple different ephemeral control tasks that

arise, are executed, and end, and each task involves a dy-

namically evolving group of controllers with dynamic state.

Furthermore, it is not sufficient to synchronize all controllers

that will be involved in a given task at the beginning of that

task, because the embedded processors are not capable of

running the number of concurrent controllers required. Such

synchronization at the beginning of the task would also be

wasted effort should the task change in midstream, such as

due to a replanning event in response to a system exception.

In order to perform the initialization, therefore, a new

controller must dynamically obtain a copy of the state of the

control group, as described in property B in Proposition 6. As

described in this property, for an observer-based controller,

the state that must be sent is not x, but rather x; x is insuf-

ficient for initializing the new controller. It must also listen

to sensor information for an interval before synchronization

through forward propagation can occur.

In the printing system, then, to synchronize a new con-

troller i to a control group j at time k, the following steps

are needed:

1) At time k−D controller i begins listening for sensor

information y

2) At time k−D a controller in the control group sends

x j(k−D) (Proposition 5) to i

3) At time k controller i sets its own variables equal to

the received state information and performs forward

propagation to construct the current state xi(k) as well

as xi(k|k−1)4) Controller i is then synchronized and begins to perform

the regular control updates.

At time k, then, the synchronization information is re-

ceived by controller i, at which time it also has available

the sensor information y(k − D − d), . . . ,y(k − 1 − d) for

which it has been listening. Note that the history state

x j(k−D− d|k−D− 1) will have all the sensor corrections

up to y(k−D− d − 1) already applied to it, because these

will all be available at k−D.

Controller i sets its internal variables xi(k−D−d|k−D−1) and ui(k−D− d), . . . ,ui(k−D− 1) equal to the equiva-

lents sent from j. Then it can perform forward propagation

to get the current state xi(k|k−1):

xi(k−D|k−D) = fmu(xi(k−D−d|k−D−1),

y(k−D−d),

ui(k−D−d), . . . ,

ui(k−D−1),k−D)

ui(k−D) = gu(xi(k−D|k−D),k−D)

xi(k−D+1|k−D) = ftu(xi(k−D|k−D),

ui(k−D),k−D)

... (19)

xi(k−1|k−1) = fmu(xi(k−1−d|k−2),

y(k−1−d), (20)

ui(k−1−d), . . . ,

ui(k−2),k−1)

ui(k−1) = gu(xi(k−1|k−1),k−1)

xi(k|k−1) = ftu(xi(k−1|k−1),

ui(k−1),k−1)

Note that as part of the last stage (20) of the forward

46

Fig. 14. Simplified statechart of the synchronization state machine (FSM).

propagation, xi(k−d|k−1) is calculated:

xi(k−1−d|k−1) =

fu(xi(k−1−d|k−2),y(k−1−d),k−1)

xi(k−d|k−1) = (21)

ftu(xi(k−1−d|k−1),ui(k−1−d),k−1−d)

As the control history ui(k − d), . . . ,ui(k − 1) is also con-

structed during the propagation, this algorithm produces all

the elements of xi(k) = x j(k), at which point it will be

synchronized with the control group.

To implement this synchronization algorithm, the module

controller is embedded in a finite state machine (FSM; see

Fig. 14) with an explicit synchronization state. When the

module controller is turned on (at a particular time), it enters

the Synch state from the Off state. During this state, the

controller waits until the synchronization time k, when it

has the control group state and sensor information described

above. It can then perform the synchronization calculations.

The FSM its state chart implementation are explained in

more detail in [17]. Once synchronization has been achieved,

the controller can progress to the Compute state, where it

performs normal control updates every Ts. From there it

advances to the Drive state when it is time for it to take

control of the actuator. Finally, at a specified time the FSM

turns itself off. This process is diagrammed in Fig. 14. The

state activation is also shown in Figs. 5 and 6.

VI. REAL-TIME SOFTWARE ISSUES

This section describes some of the networked, real-time

software developed to deploy the control and coordination

algorithms discussed above on the highly modular printer

prototype. The distributed nature of the system can be seen

in the deployment diagram in Fig. 15. The planner/scheduler

has its own processor, as does the sheet controller factory

(an entity that generates sheet controllers). Each module has

its own processor. In general, the software is designed to

support modules containing any number of nips or sensors.

The modules and the sheet controllers are connected via a

CAN-bus (Controller Area Network), and the sheet controller

<< control area network >>

MCFactoryMCFactoryMCFactory

SCFactory

<< local area network >>

Planner

Fig. 15. Deployment diagram showing the distributed control systemimplementation.

Fig. 16. Flow of sensor data through the system.

factory (and sheet controllers) communicate with the planner

over Ethernet (TCP). As an example, the complete flow of

sensor data in the system is diagrammed in Fig. 16.

We will describe here the real-time portions of the soft-

ware: the sheet controller factory and sheet controllers, and

the module controller factory and module controllers. A

simplified class diagram representing the software structure

is shown in Fig. 18. The diagram, while not containing

all the details of the software, is designed to capture the

key elements and portray the philosophy behind the design.

The sheet controller factory’s main role is to receive new

sheet itineraries from the planner and create new sheet con-

trollers corresponding to these itineraries. It also maintains

a system model of the machine configuration for the sheet

47

controllers, and passes changes in the model (received from

the modules, for example) along to the planner. The sheet

controllers themselves are responsible for sending commands

to the modules, maintaining a model of the sheet’s progress,

and subscribing to, processing, and distributing sensor and

controller state data related to the sheet.

The module controller factory resides in the module

processor. Its main job is to create module controllers in

response to new tracking commands sent by a sheet con-

troller. Multiple module controllers can exist at the same

time, though only one will be actively engaging any given

nip – the others will be preparing for upcoming sheets, for

example. Separate module controllers can also be spawned

for different nips within a module, in the generic case.

In addition, the module controller factory is responsible

for monitoring any sensors in the module and maintaining

subscriptions to those sensors (sending sensor events to a

subscribed sheet controller). It also passes along controller

state data to subscribed sheet controllers.

The roles of the sheet controller factory and the module

controller factory are diagrammed in Fig. 17. Each new sheet

itinerary from the planner causes a new sheet controller to be

created by the sheet controller factory. The sheet controller

remains active until the sheet has left the machine. It in turn

sends appropriate command messages to the relevant module

controller factories, which instantiate module controllers to

handle the sheet while it is in contact with their local nips.

The module controllers remain alive for the duration of time

that the paper is in the module. This timing information is

supplied in the command message.

A more detailed view of the software components is

portrayed in the class diagram (Fig. 18). When a new sheet

itinerary reaches the sheet controller factory, then, a new

sheet controller is spawned through the processNewPlan()

function. At each time step, the sheet controller factory runs

a mainControlLoop(), in which it cycles through all existing

sheet controllers and allows them to update their internal

model of the sheet (sheetUpdate()), send new commands

(including sensor subscriptions) to modules that are about

to engage the sheet (generateModuleMessages()), process

incoming sensor messages from the modules (processSen-

sorMsg()), send relevant sensor information to the modules

actively engaged on the sheet (createSensorUpdates()), and

forward on controller state information received (forward-

StateMsg()). The sheet controller factory maintains configu-

ration data about the system, which includes information on

different types of modules, the modules in the system, and

their connectivity. This configuration information allows the

sheet controller to correctly segment trajectory information

for the different modules and nips.

At each control cycle, the module controller factory exe-

cutes its mainControlLoop(). Within this loop it checks for

messages from sheet controllers (processMessage()), checks

its sensors and publishes any sensor events (checkSensors()

and maintainSensorSubscriptions()), notifies any active mod-

ule controller that needs to send its state out for synchroniza-

tion purposes (sendStateMsg()), allows each active module

Fig. 19. Functional groups in the sheet controller.

controller to update its state (updateController()), and outputs

any controls to its actuators (outputControls()).

The message types used by the sheet and module con-

trollers are also shown in Fig. 18. Each message has a

message type flag and a data payload. The main classes

of messages are command messages, which also include

sensor subscriptions, delete command messages, sensor event

messages (from a module controller to a sheet controller),

state announce messages (from a module controller to a sheet

controller), sensor update messages (from a sheet controller

to a module controller), state update messages (from a sheet

controller to a module controller), and configuration change

messages.

The sheet controller and module controller can be seen

to have parallel functionality, although they may be imple-

mented somewhat differently in practice: each has a sensor

and state data monitoring and processing functionality and

an observation and control functionality. The sheet controller

actions, for example, can be aggregated into two functional

groupings, as shown in Fig. 19. The sheet controller can

be viewed as having a Sensor/State Manager component

and a Sheet Observer component. The Sensor/State Manager

component receives sensor event messages from the modules,

processes them, and sends out sensor update messages to the

current coordinated control group. It also forwards control

group state data that it receives to new modules needing

to synchronize with the group. The Sheet Observer tracks

the progress of the sheet and, when a new module is being

brought into the control group, sends appropriate command

and sensor subscription messages.

The module controller processes can also be aggregated

into functional groups, depicted in Fig. 20. The Sensor/State

Manager is responsible for monitoring the module’s sensors,

detecting whether there has been a change (edge crossing),

and reporting any sensor events to sheet controllers that have

subscribed to the relevant sensor. It is also responsible for

sending a copy of an active controller’s state data to the sheet

controller when required for synchronization. The Delay

Equalizer is responsible for accepting incoming sensor and

state update messages from the sheet controller and delaying

them until the appropriate apply times. It then supplies the

sensor or state data to the relevant Nip Control element. The

third module controller functionality is, of course, running

the control algorithm and its synchronization finite state

machine. It is responsible for maintaining the correct control

state and calculating the control signals.

48

new

scFactory

sc1

sc2

planner mcFactory1 mcFactory2

mc12

mc21

mc22

mc11

new

new

cmdMsg

cmdMsg

sh2 itin

sh1 itin

new

new

newcmdMsg

cmdMsg

Fig. 17. Sequence diagram showing the instantiation of sheet controllers and module controllers by their respective factories.

Fig. 20. Functional groups in the module controller.

Statecharts (see, for example, Harel [15], Selic et al.

[26], and Samek [24]) are a useful technique for visualizing

dynamic and concurrent aspects of real-time software. In

Fig. 21, we have attempted to capture the different concurrent

tasks running on the sheet controller and module controller

along with their event-driven, message-based interaction.

Note that in our implementation these are not actually

separate threads, but are conceptually separate processes that

are activated in different ways (time-triggered or event-based,

for example).

A sheet controller starts off in the “Running” state, which

contains two orthogonal regions (two concurrent threads or

conceptual processes), separated by the dashed line. The

“SCSensor/StateMgr” process is triggered by the “sensEv-

entMsg” or “stateAnnounceMsg,” which are messages gen-

erated by the sensor/state manager in the module controller.

The “SCSensor/StateMgr” process processes a sensor mes-

sage by applying a transformation to the sensor data, gener-

ating an appropriate apply time, and then relaying the sensor

data to the nips that are currently acting on the sheet by gen-

erating a sheet controller sensor message “sensUpdateMsg,”

which is sent to the module controller over the network. This

is indicated by the “ˆ” notation in the figure. Upon receipt

of a state announcement message, the sensor/state manager

simply forwards the state information to the relevant module

controller via a “stateUpdateMsg.” The “SheetObserver”

process runs concurrently with “SCSensor/StateMgr.” It is

triggered synchronously, every control time sample, as shown

by the self-transition labeled by the event “Ts.” At each time

step, its role is to maintain an updated estimate of the sheet

position, via “sheetUpdate(),” to generate an “ERROR” if a

sensor measurement is too deviant from the expected sheet

position, and to wake up the appropriate module controllers

when the sheet position approaches their location by gener-

ating the command message “cmdMsg” event, which is sent

to the module controllers over the network. The command

messages include various data and parameters related to the

itinerary of the particular sheet being handled, which the

module controller uses for computing the control signals that

are sent to the nips. Command messages can also be sensor

subscription messages.

A module controller starts in the “OFF” state, and transi-

tions to the “ON” state upon receipt of a trajectory-specifying

“cmdMsg.” The “ON” state contains three orthogonal regions

which run simultaneously. The “MCSensor/StateManager”

process synchronously polls the sensors every control time

step and generates the event “sensEventMsg” when an edge

is detected. This message contains data which includes the

time of detection (time-stamp) and sensor identification,

and is sent to the sheet controller over the network. The

“MCSensor/StateMgr” also sends a “stateAnnounceMsg”

when required for control group synchronization purposes.

The “Delay Equalizer” process is triggered by the arrival

of a new sheet controller sensor or state update message.

49

Fig. 18. Simplified class diagram for the high-end printing system control. SC = sheet controller, MC = module controller.

It stores the message in a list. It then prepares messages

from the list whose apply time matches the current time

and identifies them as ready, thereby making it appear to

the “NipControl” process that all arriving messages have a

fixed delay (though the two different message types may have

different fixed delays). The “NipControl” process is triggered

synchronously every control time step “Ts”. It dispatches any

ready sensor or state messages to the synchronization state

machine and the PTOS control algorithm which computes

the control signals. Once the sheet is about to arrive at that

nip, the control is actually applied to the nips.

VII. RESULTS AND CONCLUSIONS

A. Experimental Results

The controller synchronization method described here has

been successfully implemented in the hypermodular high-

speed printer prototype. In the results shown here, the

controller sample time was 2ms and d and D were 5 and

6 controller time steps, respectively. The controller time step

Ts was 2ms. The maximum delay values were determined

through a combination of calculation and empirical testing

Fig. 21. Statechart showing the signal interactions between a sheetcontroller and a module controller.

50

65 66 67 68 69 70 710

0.5

1

1.5

Time (s)

Contr

ol S

ignals

(m

/s)

Fig. 22. Control signals in the printer prototype for a single sheet. Differentnips are represented in a rotating set of four colors; as they are synchronized,the plots are overlaid. Time values are from local controller time stamps.The print engine has constant-speed internal nips that are not part of thecontrol group; the empty space on the plot corresponds to where the sheetis entirely inside the print engine.

65.1 65.12 65.14 65.16 65.18 65.2 65.22 65.24 65.26 65.28 65.31.2

1.22

1.24

1.26

1.28

1.3

1.32

1.34

1.36

1.38

1.4

Time (s)

Co

ntr

ol S

ign

als

(m

/s)

Fig. 23. Closeup of the control signals. x-es are recorded control values.As the red and blue nips join the control group, their actions are exactlysynchronized.

of delay during peak network load. There were 5 CAN

buses used for control, and several additional buses used for

debugging and for some of the clock synchronization.

A sample trace of the controls generated when sending a

single sheet through the system is shown in Fig. 22, with a

closeup in Fig. 23. The path of the sheet is the one shown in

Fig. 2. Control signals for the 18 different nips on the path

are plotted, but the controllers are successfully synchronized

so the traces cannot be distinguished. The control values are

the commanded velocities for the nips. The control trace

follows a sheet as it comes out of the feeder at a nominal

highway speed of 1.25m/s, slows down to enter the print

engine, speeds up again after exiting the print engine, and

then continues out of the machine at the highway speed. The

bumps in the control signals correspond to the apply times

for the data obtained from sheets crossing sensor locations.

The reference tracking performance in the system is shown

in Fig. 24, with a closeup in Fig. 25. The tracking is quite

good, with errors within a few millimeters even in challeng-

ing points for tracking (during the speed-ups and slow-downs

on the on- and off-ramps), where the PTOS inflection-free

tracking comes into play. The successful synchronization of

the system can also be seen in the indistinguishable reference

tracks for the different module controllers. The tracking

performance of the system with respect to the high-level

planner timepoints is shown for a set of twenty sheets in

Fig. 26.

64 65 66 67 68 69 70 710

0.5

1

1.5

2

2.5

3

3.5

Time (s)

Dis

tan

ce

(m

)

Sensor data

Controller reference

Fig. 24. Reference tracking in the module controllers for a single sheet.Different nips are represented in a rotating set of four colors; as they aresynchronized, the reference plots are overlaid. Time values are from localcontroller time stamps.

65 65.1 65.2 65.3 65.4 65.5 65.6 65.7 65.80.2

0.3

0.4

0.5

0.6

0.7

0.8

Time (s)D

ista

nce

(m

)

Sensor data

Controller reference

Fig. 25. Closeup of the reference tracking, showing the reference reset.

B. Conclusion

In this tutorial paper, we have discussed challenges, solu-

tions, and lessons learned from the design of a hierarchical

distributed networked control approach for a highly reconfig-

urable printing system. We have described: the hardware plat-

form, along with the physical parameters and performance

requirements; the system dymanics along with a hierachical

control approach, including planning, coordination and time-

optimal based feedback/feedforward control; synchronization

and handoff mechanisms for distributed deployment; realtime

embedded software architecture; and real data collected from

the system that was built at PARC.

Perhaps some of the most important general lessons that

we have learned from this experience are quite widely

applicable. The first is due to Einstien: “Everything should

be made as simple as possible - but not simpler”. This is

particularly true when it comes to system integration, where

any unnecessary design complexity generally incurs far more

effort and delays in system integration than expected. Using

a more complex design should be explored only when the

simpler alternatives have been proven to deliver insufficient

performance. The second lesson is: “Start with feasibility;

strive for optimality”. Again, we have found it extremely

helpful to get a working prototype with full functionality up

and running as soon as possible, even if the performance is

suboptimal. The point here is not to encourage “hacking”,

or discourage rigorous analytical thinking. Rather, the point

is that, when designing a new system for the first time,

51

48 49 50 51 52 53 54 55 56 57 58 590

0.5

1

1.5

2

2.5

3

Time (s)

Dis

tan

ce

(m

)

Sensor data

Planner timepoints

Fig. 26. Tracking performance with respect to planner time points, fortwenty sheets. Line style indicates which print engine a sheet used: blue/red= top/bottom, solid/dashed = left/right.

until the system is up and running, the relative importance

of different design problems is sometimes very difficult to

assess. The real system can help to put these problems in

proper perspective. New simulation and verification tools are

making things easier all the time, but in the end, it seems

that some iteration is always necessary to optimize the final

product.

In closing, we hope that this material, along with that

presented in the supporting three sessions, will demonstrate

that modern printing systems are some of the most sophisti-

cated devices in use today, which require a truly remarkable

range of engineering, and in particular, control techniques,

to enable them to deliver the level of performance that we

enjoy today.

VIII. ACKNOWLEDGMENTS

We would like to thank all of our colleagues at PARC and

at Xerox for their collaboration on this project.

REFERENCES

[1] K.J. Astrom and B. Wittenmark. Computer Controlled Systems.Prentice Hall, 1997.

[2] M. Athans and P. Falb. Optimal Control. McGraw Hill, 1966.[3] A. Bemporad, F. Borrelli, and M. Morari. Model predictive control

based on linear programming - the explicit solution. IEEE Trans. Aut.

Contr., 2002.[4] J. Bobrow, S. Dubowsky, and J. Gibson. Time-optimal control of

robotic manipulators along specified paths. Int. J. Robotic Research,1985.

[5] F. Borrelli. Constrained Optimal Control of Linear and Hybrid

Systems. Springer, 2004.[6] A.E. Bryson and Y.-C. Ho. Applied Optimal Control. Academic Press,

1975.[7] C.-L. Chen and G. Chiu. Incorporating human visual model and spatial

sampling in banding artifact reduction. In Proceedings of the American

Control Conference, Boston, MA, June 2004.[8] H. Choset, K. Lynch, S. Hutchinson, G. Kantor, W. Burgard,

L. Kavraki, and S. Thrun. Principles of Robot Motion. MIT Press,2005.

[9] J. Craig. Introduction to Robotics: Mechanics and Control. PrenticeHall, 2004.

[10] L. S. Crawford, H. Hindi, R. Zhou, and Dan Larner. Synchronizedcontrol in a large-scale networked distributed printing system. In Pro-

ceedings of the International Conference on Robotics and Automation,2009.

[11] G. Franklin, D. Powell, and M. Workman. Digital Control of Dynamic

Systems. Addison Wesley, 1998.

[12] M. P. J. Fromherz, L. S. Crawford, and H. A. Hindi. Coordinatedcontrol for highly reconfigurable systems. In Hybrid Systems: Com-

putation and Control (HSCC), Zurich, Switzerland, 2005. Springer-Verlag.

[13] M. Gergeleit and H. Streich. Implementing a distributed high-resolution real-time clock using the CAN-bus. In 1st International

CAN-Conference 94,, Mainz, Germany, September 1994.[14] E. Hamby and E. Gross. A control-oriented survey of xerographic

systems: basic concepts to new frontiers. In Proceedings of the

American Control Conference, Boston, MA, June 2004.[15] D. Harel. Statecharts: a visual formalism for complex systems. Science

of Computer Programming, 8, 1987.[16] H. Hindi, L. Crawford, R. Zhou, and C. Eldershaw. Efficient waypoint

tracking hybrid controllers for double integrators using classical timeoptimal control. In Proceedings of the IEEE Conference on Decision

and Control, December 2008.[17] H. A. Hindi, L. S. Crawford, and M. P. J. Fromherz. Synchronization

of state based control processes with delayed and asynchronousmeasurements. In IEEE Conf. Decision and Control, December 2005.

[18] M. Krucinski, C. Cloet, R. Horowitz, and M. Tomizuka. A mecha-tronics approach to copier paperpath control. In Proceedings of the

first IFAC conference on mechatronics systems, Darmstadt, Germany,September 2000.

[19] F. Lewis and V. Syrmos. Optimal Control. Wiley, 1995.[20] P. Li, T. Sim, and D. Lee. Time sequential sampling and reconstruction

of tone and color reproduction functions for xerographic printing. InProceedings of the American Control Conference, Boston, MA, June2004.

[21] B. Lincoln and B. Bernhardsson. Optimal control over networks withlong random delays. In Proceedings of the International Symposium

on Mathematical Theory of Networks and Systems, 2000.[22] R. Luck and A. Ray. An observer-based compensator for distributed

delays. Automatica, 26(5):903 – 908, May 1990.[23] J. Nilsson. Real-time control systems with delays, 1998.[24] M. Samek. Practical statecharts in C/C++. CMP Books, 2002.[25] R. Sanfelice, R. Goebel, and A. Teel. Invariance principles for hybrid

systems with connections to detectability and asymptotic stability.IEEE Trans. Aut. Contr., 2007.

[26] B. Selic, G. Gullekson, and P. Ward. Real-Time Object Oriented

Modeling. Wiley, 1994.[27] A. S. Tanenbaum and M. van Steen. Distributed Systems: Principles

and Paradigms. Prentice Hall, New Jersey, 2002.[28] W.M. Wonham. Linear Multivariable Control: A geometric Approach.

Springer-Verlag, 1985.

52