ORIGINAL ARTICLE

Claudio S. Pinhanez Æ Aaron F. Bobick

Interval scripts: a programming paradigm for interactiveenvironments and agents

Received: 1 May 2002 / Accepted: 5 October 2002� Springer-Verlag London Limited 2003

Abstract In this paper we present interval scripts, a newparadigm for the programming of interactive environ-ments and computer characters. In this paradigm,actions and states of the users and the system computa-tional agents are associated with temporal intervals.Programming is accomplished by establishing temporalrelationships as constraints between the intervals. Unlikeprevious temporal constraint-based programming lan-guages, we employ a strong temporal algebra based inAllen’s interval algebra with the ability to expressmutually exclusive intervals and to define complex tem-poral structures. To avoid the typical computationalcomplexity of strong temporal algebras we propose amethod, PNF propagation, that projects the networkimplicit in the program into a simpler, 3-valued (past,now, future) networkwhere constraint propagation can beconservatively approximated in linear time. The intervalscripts paradigm is the basis of ISL, or Interval ScriptsLanguage, that was used to build three large-scale,computer-vision-based interactive installations withcomplex interactive dramatic structures. The success inimplementing these projects provides evidence that theinterval scripts paradigm is a powerful and expressiveprogramming method for interactive environments.

Keywords Interactive spaces Æ Programming paradigms ÆProgramming with constraints Æ System architecture ÆTemporal reasoning Æ Ubiquitous computing

1 Introduction

Recent years have seen an increasing interest in the de-velopment of interactive environments where users in-teract with devices, data, images, computational agents,and computer characters. Such environments can bevirtual spaces as it is the case of Virtual Reality (VR)systems, or real physical spaces with embedded com-puting devices and control. Examples of the latter in-clude location-based entertainment spaces [1–4],interactive offices [5–7], classrooms [8] and homes [9, 10].

However, little attention has been paid to the specialrequirements imposed by interactive environments onprogramming methods. In particular, in highly immer-sive situations both the actions and states of the envi-ronment and its computational agents, as well as theactions of the users, are not instantaneous events, buttake time to be performed. This makes difficult the use ofthe traditional event-based programming paradigmcommonly employed in desktop interaction.

Although the problem of system actions with non-zero duration has been the object of research in themultimedia community, interactive environments extendsignificantly the problem since they include situationswhere the user’s actions also extend through periods oftime. Unlike desktop systems where the interaction oc-curs mostly through mouse clicks, in interactive envi-ronments the users talk, walk, make complex gestures,and interact with objects, computational agents, com-puter characters, and other users. However, the pro-gramming systems for interactive environmentsproposed so far (e.g. [11, 12]) use simple paradigms thatdo not provide adequate ways to handle this kind oflong-duration interactivity.

Let us examine an example to better characterise theissues involved in programming an interactive environ-ment. Consider a simple interactive environment in amuseum context, composed of three different areas A, Band C, where pre-recorded tapes have to be played

Pers Ubiquit Comput (2003) 7: 1–21DOI 10.1007/s00779-002-209-4

Claudio S. Pinhanez (&)IBM Research, T.J. Watson,19 Skyline Drive, Hawthorne,NY 10532, USAe-mail: pinhanez@us.ibm.com

Aaron F. Bobick (&)Georgia Institute of Technology,GVU Center, 801 Atlantic Dr.,Atlanta, GA 30332, USAe-mail: bobick@cc.gatech.edu

whenever a user enters the corresponding area. Let’s alsoassume that the system is able to differentiate amongusers and that the tapes cannot be played simultaneouslybecause the sound overlap makes them unintelligible.Figure 1 shows some typical difficulties to the pro-gramming of such system. In instant t0, user 1 entersspace A, triggering the playing of tape A. Notice thatthere is a delay in the start of the tape. Most important,however, if we do not want the tape to play again whileuser 1 is in the area A (for instance, after instant t3), thenthe control system cannot be purely reactive, i.e. basedon a simple event structure such as ‘‘if user is in area A,play tape A’’, because that event would be repeatedlyactivated by the presence of user 1 in area A. Here, thecontrol program needs some way to represent and accessnotions of continuous states and interaction history inorder to activate an action.

When user 2 enters space B in instant t1, the systemnormally would start playing tape B. However, sincetape A is playing, the start of tape has to be delayed,because the two actions, as described before, are mutu-ally exclusive. However, since tape A lasts longer thanthe presence of user 2 in area B, the playing of tape B hasto be, ultimately, ignored. The moving of user 2 to areaC, in instant t2, however, does not initiate the playing ofthe tape C either. Only after tape A ends, at t3, it ispossible to start tape C. Instant t5 illustrates anothercommon problem: although tape C is playing, there is nouser in area C. This is a typical situation where it wouldbe desirable to have ways to stop an action abruptly, inthis case by triggering a fadeout or similar effect. Finally,when tape C ends on instant t6, the system is faced with achoice between equivalent but mutually exclusively ac-tions: it can either play tape A or tape B, but not both.

It is easy to see that similar problems arise also indifferent scenarios of interactive environments. Suppose,for instance, a home environment for a visually impairedperson where two computer agents, agent 1 and agent 2,provide help with communication with other people andwith dangerous situations, respectively. Looking at thesame Fig. 1, agent 1 detects at instant t0 that the personis in the environment, and decides to deliver a messageleft for her (tape A). At instant t1, agent 2 sees apotentially dangerous situation where an advisory mes-sage could be helpful. Like in the previous example,there is a conflict for resources here, since two speechactions cannot happen at the same time. It is straight-forward to see how the situation described in Fig. 1develops in this scenario with a warning (a reaction to

situation B at t1) becoming obsolete in t2, and the alert ofa communication attempt (for instance, a call from arelative) being preempted at t5 because of a warningrelated to a dangerous situation (tape C).

As we see in the above examples, a programmingparadigm for interactive environments has to addresscomplex temporal structures that are a result of users’and computer actions and states having duration. Al-though it is theoretically possible to represent suchtemporal structures using simple models such as statemachines or event-loop systems, this approach requiresthe enumeration of all possible combinations of states orevents, creating programs that are hard to write, debug,and maintain. Moreover, since such systems require thateach combination of states to be represented as a singlemachine state or event, the addition of new actions,sensors, agents, or users often causes an exponentialincrease in the number of machine states or events. Thisis a direct consequence of having mutually exclusiveparallel actions or states (see Pinhanez [13]). Suchproblems have been consistently found by the authors inthe development of interactive environments [2, 4, 5, 14].

This paper proposes a method to program interactiveenvironments based on temporal constraints. Called in-terval scripts, this paradigm encapsulates actions andstates in temporal intervals constrained by the temporalprimitives of Allen’s algebra [15]. Our objective is toprovide the programmer of an interactive environmentwith a tool that combines expressiveness and simplicityand, at the same time, produces control code that can berun in real time. Among the desired programming fea-tures, we include in interval scripts not only facilities forthe simple handling of time durations, but also featuressuch as the definition of mutually exclusive actions, theability to impose restrictions on what can happen (neg-ative scripting), easy encapsulation of actions, basichandling of interaction history, and automatic recoveryfrom errors and unexpected situations.

To provide these programming facilities, an intervalscript contains descriptions of how to initiate and stopeach action of the interactive environment or its agentsand how to determine the occurrence of actions andstates of the users, environment states, and the states ofcomputational agents. The control of activation of anaction is determined by verifying the current state of theaction, comparing it with the desired state according tothe constraints with other actions, and issuing com-mands to start and stop actions as needed. Also, as wedescribe in detail later, interval scripts de-couple the

Fig. 1 Examples of difficultieswhen programming interactiveenvironments

2

actual state of an action from its desired state, allowingfor the introduction of methods for recovery from sensorerrors and increasing the robustness of the system.

Traditional constraint propagation methods cannotbe used to run the interval algebra networks associatedto the interval scripts due to the typical size of the net-works and the requirement of real-time computation[16]. To solve this problem, our system employs a novelmethod for constraint propagation in such networksthat overcomes these limitations by projecting the net-work implicit in the script into a simpler, 3-valued (past,now, future) PNF-network, where constraint propagationcan be approximately computed in linear time (initiallyproposed in Pinhanez et al. [17])

This paper starts by reviewing the current program-ming paradigms and identifying their shortcomings. InSect. 3 we introduce the interval scripts paradigmthrough some simple examples. The core of the paper isthe description of the PNF theory needed to allow theefficient computation of temporal constraint propaga-tion algorithms, and of the run-time engine architecturethat uses it, as described in Sect. 4 and 5. An imple-mentation of interval scripts paradigm into a program-ming language called Interval Scripts Language, or ISL,is described in Sect. 6, together with examples of its use.Section 7 explores the three concrete cases of interactiveenvironments that have been built using the paradigmand in Sect. 8 we discuss future directions in ourresearch.

2 Problems with current programming techniques

The idea of interval scripts was spawned by our dis-satisfaction with the tools for the integration of I/Odevices and software modules in interactive environ-ments and for the control of computer agents. Asdescribed by Cohen, systems to control interaction tendeasily to become ‘‘big messy C program(s)’’ ([18],Fig. 2). Also, from our experience with The KidsRoom[2], it became clear that one of the major hurdles to thedevelopment of interesting and engaging interactiveenvironments is that the complexity of the controlprogram grows faster than the complexity of the pro-grammed interaction.

For example, The KidsRoom was programmed usingan event-loop paradigm [2]. However, during thedevelopment of The KidsRoom we encountered manysituations where unanticipated and undesirable co-occurrences of events disrupted considerably the inter-action. A typical case, similar to the one described in theintroduction, was when multiple characters were trig-gered to talk or act by different events. The problem isthat the characters, in many trials, ended up beingtriggered almost simultaneously, creating unintelligibleoverlapped speech or action. Since the environment useda control structure based in event-loops, it becamenecessary to examine each possible co-occurrence and toexplicitly program how to handle every individual case.

However, for each new action or character added to theenvironment, it was necessary to explore in the code theinteraction between its actions with every action of othercharacters. In other words, the result was a de factoexponential increase in the number of different situa-tions to be handled by the programmer.

2.1 Finite-state machines and event loops

The most common technique used to program andcontrol interactive applications is to describe the inter-action through finite-state machines. This is the case ofone of the most popular languages for the developing ofmultimedia software, Macromedia Director’s Lingo [19].In Lingo the interaction is described through the han-dling of events whose context is associated with specificparts of the animation. There are no provisions to re-member and reason about the history of the interactionand the management of story lines. The same situationoccurs with Max [20], a popular language for control ofmusic devices (see Roads [21]) and adopted in some in-teractive art installations [22]

Videogames are traditionally implemented throughsimilar event-loop techniques [23]. To represent the in-teraction history, the only resort is to use state descrip-tors whose maintenance tends to become a burden as thecomplexity increases. Most of all, a fundamental prob-lem with the finite state model is that it lacks appropriateways to represent the duration and complexity of humanaction, computational agents, and interactive environ-ments: hidden in the structure is an assumption thatactions and occurrences are pinpoint-like events in time(coming from the typical point-and-click interfaces forwhich those languages are designed).

2.2 Constraint-based programming languages

The difficulties created by the finite-state based modelhave sparked a debate in the multimedia research com-munity concerning the applicability of constraint-basedprogramming (starting with the works of Buchanan andZellweger [24] and Hamakawa and Rekimoto [25]) ver-sus procedural descriptions such as state machines (forexample, see van Rossum et al. [26]). In general, it isbelieved that constraint-based languages are harder tolearn but more robust and expressive as discussed inVazirgiannis et al. [27].

For example, Bailey et al. [28] defined a constraint-based toolkit, Nsync, for constraint-based programmingof multimedia interfaces that uses a run-time schedulerbased on a very simple temporal algebra. The simplicityof the temporal model and, in particular, its inability torepresent non-acyclic structures, is a major shortcomingof Nsync and similar systems such as Madeus [29],CHIMP [30], ISIS [31] and TIEMPO [32].

Andre and Rist [33] have built a system called PPPthat designs multimedia presentations based on

3

descriptions of the pre-conditions of the media actions,their consequences and on the definition of temporalconstraints. The PPP system uses a strong, hybridtemporal algebra that combines intervals and points asits primitives (based on Kautz and Ladkin’s [34] work).The descriptions of media actions are used to plan amultimedia presentation that achieves a given goal. Themultimedia presentation is represented as a constraintnetwork. During run-time, the scheduler has to buildand choose among all the possible instances of the plans.However, since Kautz and Ladkin’s propagation tech-niques are exponential in time, and the number of pos-sible plans can be exponentially large, the applicabilityof PPP is restricted to simple cases with limited inter-action (otherwise the number of possible plans increasestoo quickly).

2.3 Programming interactive environments

Starting with the pioneer work of Myron Krueger [35],the interest in building interactive environments, hassubstantially grown in the last decades (see Bedersonand Druin [36] for a review). In the case of virtual re-ality, it seems that the user-centred and exploratorynature of most interfaces facilitates the programming ofthe interface by using state machines and event loops,using languages such as MR [11] and Alice [12]. Thelatter was used in Disney’s Aladdin ride (described inPausch et al. [1]). It is a language that allows rapidprototyping but can hardly describe any non-trivialtemporal structure.

Cohen [18] proposed a distributed control structure,the Scatterbrain. In this paradigm, used in the control ofa sensor-loaded room, knowledge and behaviour is dis-persed among multiple agents that negotiate amongthemselves. Although an appealing model, the use ofmultiple agents without centralised control makes au-thoring extremely difficult in practice. At the oppositeend of the spectrum, Dey [37] proposes a framework forthe prototyping of context-aware environments. Bothproposals lack models of temporal control capable ofefficiently handling even modestly complex temporalstructures such as the ones described in the introductionsection.

3 The interval scripts paradigm

In traditional desktop-based interfaces and programs,system anduser actions are assumed to have no significantduration, except in some cases of multimedia program-ming. In the same manner, traditional programming isbased on the concept that the states of all variables remainthe same as long as no event occurs.Whenan event occurs,a piece of program is executed, normally assuming ‘in-stantaneous’ response, and then a new state is reached. Inother words, states are a-temporal, i.e. they last as long asno new detected event happens.

As described in the example of the introduction, in-teractive environments and agents perform actions thatoften take a non-trivial amount of time, such as playinga sound file, showing a character animation, or closing adoor. Similarly, user actions many times also have non-zero duration, such as when performing a gesture,walking around the space, or saying a sentence to aspeech interface. These observations drove us to design aprogramming paradigm where all elements of a pro-gram, including its states and actions, are assumed tohave non-zero duration.

The fundamental concept of the interval scripts par-adigm is that all actions and states are associated withthe temporal interval in which they occur. However, theactual beginning and end time of the intervals are notpart of the script. Instead, the script contains a de-scription of the temporal relationships or constraints thatthe intervals should satisfy during run-time. These con-straints are enforced by a run-time engine that examinesthe current states of all intervals, and then coordinatesthe system’s reaction to ensure that the interval’s statesrespect the constraints.

For example, suppose we want to describe the situa-tion where a door opens whenever the user is in front of it.To do this with interval scripts, we associate the userpresence in front of the door to an interval A and theaction of opening of the door to another interval B. Wethen put a temporal constraint stating that the interval oftime of A always has the same start time as the interval BDuring run-time, the interval scripts engine monitors thepresence of the user in front of the door.When it sees him,the engine detects that, according to the constraint, bothintervals should be happening at the same time and triesto open the door in an attempt to satisfy the constraint.

Developing an interface or interactive system withinterval scripts is a classical case of programming byconstraints as discussed in Sect.2. However, the idea ofusing temporal constraints to describe interaction ininteractive environments initially appeared in Pinhanezet al. [17], in a different formulation from the one de-scribed in this paper. That work drew from a techniquefor representing the temporal structure of human actionsby Pinhanez and Bobick [38]. The formalism and theorypresented here considerably improve the syntax and se-mantics of the paradigm, as well as introduce a muchmore consistent, efficient, and robust run-time enginethan in those works, as we will see in the next sections.

3.1 Allen’s interval algebra

To model the time relationships between two actions, aprogram in interval scripts employs the interval algebraproposed by Allen [15]. The representation of temporalconstraints in Allen’s algebra is based on the 13 possibleprimitive relationships between two intervals as shown inFig. 2. Given two actions or states in the real world, alltheir possible time relationships can always be describedby a disjunction of those primitive time relationships.

4

For instance, in the preceding example, the relation-ship A i-start B correctly expresses the temporal con-straint between the presence of the user (interval A) andthe time needed to open the door (interval B), if weassume the user waits for the door to open. In practice,however, it is hard to determine whether the user willleave the door area before the end of the action ofopening the door, so the relationship between A and Bis better described by the disjunction start OR equal ORi-start, i.e. that the two intervals start at the same time.Of course, in any actual occurrence of the intervals,only one of the disjunctive elements actually happens.

To simplify the manipulation of the intervals andtheir constraints, a set of actions and states and theirrespective temporal constraints can be represented by anetwork in which each action or state is associated witha node and the temporal constraints between them arerepresented by arcs. The networks composed of suchnodes and disjunctions of Allen’s primitive relationshipsas arcs are called interval algebra networks, or simply,IA-networks [15]. It is important to notice that the nodesare, in fact, variables that can assume any temporal in-terval representing the actual interval of time of when anaction or state occurs.

3.2 Strong interval algebras

Allen’s algebra allows the expression of mutually ex-clusive intervals. For instance, to state that a computeragent does not perform actions C and D at the same timewe simply constrain the relation between the intervals Cand D to be meet OR i-meet. That is, in every occur-rence of the intervals, either C comes before D or afterD, with no overlap.

The ability of expressing mutually exclusive intervalsdefines different classes of temporal algebras [39]. Ingeneral, algebras without mutual exclusiveness allowfast constraint satisfaction but are not expressive [16].However, in our approach we are able to employ Allen’sstrong algebra in a real time system because we havedeveloped a fast method to compute approximations of

the values that satisfy the constraints as described laterin this paper.

Andre and Rits [33] have also used a strong temporalalgebra to script multimedia presentations. Their workemploys a more expressive temporal algebra that com-bines Allen’s [15] and Villain’s [16] proposals as sug-gested by Kautz and Ladkin [34]. However, constraintpropagation in this temporal algebra is basicallyO n2 eþ n3

� �� �, where n is the number of nodes and e is

the time required to compute the minimal network of theIA-network; typically, e is exponential in the number ofnodes, O eð Þ ¼ O 2nð Þ, making the whole system basicallyO n22n þ n5� �

¼ O 2nð Þ. Moreover, their system requiresthe enumeration in computer memory of all possibletemporal orders of actions implied by the constrainednetwork, which can grow exponentially, especially in thecase of multiple and frequent user interaction.

3.3 Example of programming withtemporal constraints

To understand how to use interval algebras in interac-tion programming, let us examine an example from thescript of one of our interactive environments, the artinstallation called It described in Sect.7. In that instal-lation a Computer-Graphics (CG) object that look likesa photographic camera, here referred to as camobject,appears on a big screen and interacts with the user in theenvironment.

Suppose we want to describe a situation wherecamobject moves front, that is, moves forward towardsthe user, produces a clicking sound (as it was taking apicture), and then moves back to its previous position.The first step is to associate each of these three actions tonodes in an IA-network, and name them, respectively,camobject moves front, camobject clicks and camobjectmoves back. To program the desired sequencing of thethree actions, we define meet constraints between eachconsecutive pairs:

‘‘camobject moves front’’ meet ‘‘camobject clicks’’‘‘camobject clicks’’ meet ‘‘camobject moves back’’

Figure 3 shows the desired relation between the threeactions. The first constraint states that the action cam-object moves front should meet with the action camobjectclicks, i.e. the latter action should start immediately afterthe end of the former. The second constraint establishesa similar constraint between the actions camobject clicksand camobject moves back.

Fig. 3 Diagram of temporal constraints showing a sequence of threeintervals

Fig. 2 Allen’s 13 primitive relationships between two time intervals

5

Now, suppose that we want to define an actioncamobject takes a picture that is composed of these threeintervals. This could be accomplished by the followingconstraints (see also Fig. 4):

‘‘camobject moves front’’ start ‘‘camobject takes apicture’’

‘‘camobject clicks’’ during ‘‘camobject takes a picture’’‘‘camobject moves back’’ finish ‘‘camobject takes apicture’’

Finally, if we want the whole sequence to start when theuser makes a pose to the camera-like object (as requiredby the art installation environment), a composed dis-junctive constraint can assure that whenever the userstarts posing, the sequence for taking pictures is acti-vated (see Fig. 4):

‘‘user makes a pose’’ start OR equal OR i-start‘‘camobject takes a picture’’

In this case the disjunction start OR equal OR i-start isthe way to impose the constraint that the action and thesensor have the same starting time without requestingany particular configuration of the ending time. In otherwords, although ‘‘user makes a pose’’ and ‘‘camobjecttakes a picture’’ are supposed to start together, either ofthem can finish first without violating this constraint.

3.4 Example of the use of mutuallyexclusive constraints

Suppose now that we do not want to allow the taking ofpictures if the user is moving in the environment. This isan example of mutually exclusive intervals mentioned

previously. First, as above, we assume that a node in theIA-network called user is moving has been associatedwith the moving state of the user as detected, for in-stance, by a sensing device. To program that the ‘click-ing’ action should not happen if the user is moving, wecan simply enforce the following constraint:

‘‘camobject takes a picture’’ before OR i-before ‘‘user ismoving’’

Figure 5 shows four possible occurrences of the sensorassociated with user is moving. Cases (1) and (2), showsituations where it happens respectively before or aftercamobject takes a picture, and therefore are compatiblewith the occurrence of this sensed activity. However,cases (3) and (4) show hypothetical situations where useris moving overlaps with camobject takes a picture, andthus it is not compatible.

Our goal is to design a run-time engine that preventsthe occurrence of case (3) by not allowing the camobjecttakes a picture action to start while user is moving is notfinished; and by immediately (maybe prematurely) fin-ishing camobject takes a picture if the user starts movingas in case (4). To understand better how this is accom-plished, it is necessary to examine how the run-timeengine actually works, as we will describe in Sect.4 and 5.

3.5 The case for Allen’s algebra

As can be seen from the above examples, there areseveral reasons to use Allen’s algebra to describe rela-tionships between actions as the programming paradigmfor interactive environments. First, no explicit mentionof the interval duration or specification of relations be-tween the intervals’ starting and endings are required.Secondly, the notion of disjunction of interval relation-ships can be used to declare multiple paths and inter-actions in a story. Third, it is possible to declaremutually exclusive intervals that describe actions andstates whose co-occurrence is not desirable.

A fourth reason to use Allen’s algebra, not discussedhere, is that it is only necessary for the programmer todeclare the relevant constraints. By applying Allen’s pathconsistency algorithm [15] to the IA-network associatedwith the script, it is possible to generate a more con-strained version of the network that contains all andonly the constraints that are consistent with time. Fifth,it is possible to determine in linear time which compu-tational agents’ and characters’ and environment’sreactions to trigger in response to users’ actions by

Fig. 4 Diagram of temporal constraints showing an action composedof a sequence of three others and a sensor that start at the same timeas the composed action

Fig. 5 Possible occurrences ofuser is moving

6

properly propagating occurrence information from onenode of the IA-network to the others, using the PNFpropagation method described next.

4 A method for fast constraint propagationin strong algebras

Programming with interval scripts involves the associ-ation of actions and world states of an interactiveenvironment to a collection of nodes in an IA-networkwith temporal constraints between them. However, inorder to control an interactive environment or agentbased on such descriptions, it is necessary to employ arun-time engine that examines which intervals arehappening, considers the constraints between them, andcontrols the start and stop of intervals as needed toenforce the constraints. To perform such actions, it isnecessary to use a process known as constraint propa-gation. In general, performing constraint propagationin an IA-network is exponential (as proved in Vilainand Kautz [16]). We overcome this problem by using asimplification of constraint propagation in IA-networkscalled PNF propagation, as described in this section.

4.1 Minimal domains of IA-networks

As mentioned above, the collection of temporal con-straints between nodes is represented by an interval al-gebra network [15], which is a network where the nodescorrespond to variables on temporal intervals and thetemporal constraints are links between them. Figure 6displays the IA-network associated with the three nodesconnected through meet constraints of the camobjectexample described before.

Traditional constraint satisfaction in interval algebranetworks tries to determine for each network node thesets of time intervals (pairs of real numbers describingsegments of the time line) that are compatible with theconstraints given the known occurrence of some inter-vals. An assignment of time intervals that satisfies theconstraints is called a solution of the network. Given anetwork node, the set of all time intervals for that nodethat belongs to at least one solution is called the minimaldomain of the node. Notice that if there is at least oneinterval in the minimal domain of 2a node that contains

a given instant of time then the node can be happening atthat time. If every time interval of the minimal domainof a node contain that given instant of time, then thenode must be happening at that given moment.

This observation constitutes the basic principle of ourrun-time engine. However, direct constraint propagationis NP-hard due to the combinatorial explosion involvedin the manipulation of sets of intervals (see Vilain andKautz [16]). To overcome this difficulty we devise anoptimisation scheme called PNF propagation comprisingtwo mechanisms, PNF restriction and time expansion.

4.2 PNF restriction

The idea of simplifying interval algebra networks wasfirst proposed by Pinhanez and Bobick [40], in a workmotivated by the problem of action recognition in acomputer vision system. The key idea is that for controland recognition purposes there is little informationcoming from the duration of the nodes. Instead, thefocus in such situations is to simply determine if an in-terval has happened (past), is happening (now), or maystill to happen (future). Based on this criterion, it ispossible to project an interval network into a similarnetwork where the nodes corresponding to the intervalscan assume only one of the three symbols, past, now, orfuture – a PNF-network.

However, in real systems it is very difficult to deter-mine exactly in which of these three states a node is. Forinstance, suppose a node that is associated to a binarysensor such as a light switch which has no mechanism tostore the past activation history. If the sensor is on, it iseasy to see that we can represent its state by associatingthe symbol now to the node. However, if the sensor is off,it is impossible for the sensor to locally determinewhether the switch was turned on in the past, or if it hasnever been switched on before, that is, that it may be onin the future. To represent such situations where infor-mation about the actual occurrence of an action or stateassociated to a node is not available, we can assignnodes with disjunctions of any of three basic symbols.For instance, the state of an off switch is easily repre-sented by the disjunctive set fpast; futureg.

Let us formalise now the concepts introduced before.To simplify notation, we start by defining the set M ofall subsets of fpast; now; futureg

(

( )

) ( )

Fig. 6 Interval algebra networkassociated with two meet con-straints and an example of PNFrestriction

7

M ¼ f[; fpastg; fnowg; ffutureg; fpast; nowg;fnow; futureg; fpast; futureg; fpast; now; futuregg

whose elements, called PNF-values, are abbreviated as

M ¼ f[; P ;N ; F ; PN ;NF ; PF ; PNF gAlso, we call any tuple W ¼ W1;W2; . . . ;Wnð Þ of PNF-values associated to the n nodes of a PNF-network acomponent domain of the PNF-network, where eachWi 2 M , and W � M �M � . . . M ¼ Mn ¼ U . Since eachelement of a component domain represents possiblestates of its associated node, we call each Wi the PNF-state of the node i.

It is easy to see that, given a primitive temporalrelationship r between two nodes A and B of a PNF-network, the admissible states of the two nodes mustsatisfy the constraints depicted in table 1. For instance,if A is happening now (N state), and A meet B, then Bcan only happen in the future (F state, as highlighted inTable 1). The construction of the table is formally jus-tified in Pinhanez [13], but it should be noticed that itcorresponds to our intuitive notions of past, now, andfuture.

If the temporal constraint is a disjunction of primitivetemporal primitives, we simply take the union of thePNF-values corresponding to the primitives. For in-stance, suppose A and B are mutually exclusive, A meetOR i-meet B, and A is happening now. Table 1 yieldsthat for the meet constraint, B may happen in the future(F), while for the i-meet constraint B must have alreadyhappened (P). By taking the union of the two constraintsin the disjunction, P [ F ¼ PF , we determine that theactual state of B must belong to PF, and therefore, thatB is not happening now.

On the other hand, if the node A is associated to anon-singleton PNF-value (i.e. PF, PN, NF, or PNF), it isstraightforward to see that the possible values for B aresimply the union of the admissible values for each ele-ment of the set associate with A. For instance, if A finishB and the PNF-state of A is PN, then the PNF state of Bis the PNF-value PN ¼ P [ N .

Given an initial set of PNF-values for each node rep-resented by a component domain W ¼ W1;W2; . . . ;Wnð Þ,we define a solution of a PNF-network under W to beany assignment of one of the basic past; now; futuref gvalues to each node i of the network in which each as-signed value belongs to Wi and that each pair of nodessatisfies the constraints of Table 1. We call the minimaldomain of a node under W as the set of all values of anode that belong to at least one solution under W.Notice that the minimal domain of a node is also a PNF-value. Similarly, the minimal domain of a PNF-networkunder W is the component domain composed of theminimal domain of each node of the network.

Let us then define a function, the restriction of a PNF-network under W, noted R Wð Þ, that computes the min-imal domain of the network under W. That is, therestriction of a PNF-network eliminates all the valuesfrom each component of W that are incompatible withat least one solution under W. Notice that the restric-tion of a component domain W is also a componentdomain, R Wð Þ � U , and that restriction is maximal, i.e.R R Wð Þð Þ ¼ R Wð Þ. Notice that if the minimal domain ofany node is empty, i.e. Wi ¼ [ for any 1 � i � n, thenthere are no solutions for the network.

Figure 6 also shows a simple example of PNF re-striction. The initial set of values for each node is shownbetween parenthesis and the values after restriction ap-pears on the right side of the arrows. In this case, weassume that it is known that the camobject clicks actionis happening (N). Since this action must happen aftercamobject moves front because of the meet constraint,the only possible state for the later node is past (P), asshown in Fig 6. Similarly, camobject moves back cannothave happened because it must follow camobject clicks,and therefore its possible state is restricted to future (F).

Computing the exact minimal domain of a PNF-network is, however, still a NP-hard problem (seeDechter [41]). Instead, we employ in our applications aconservative approximation to the minimal domainbased on the computation of the arc-consistency (asproposed by Mackworth [42]) of the PNF-network.The big advantage is that arc-consistency is O cð Þ inthe number of constraints c, that is, at most O n2

� �in the

number of nodes n.The procedure in Fig. 7 shows an algorithm that

computes the maximal arc-consistent domain of ann-node PNF-network under a component domainW ¼ W1;W2; . . . ;Wnð Þ. This is a version of the arc-con-sistency algorithm AC-2 proposed in Mackworth [42]and adapted to the component domain notation. Thealgorithm uses the function / which given a PNF-valueand a set of primitive relations, returns a PNF-valuethat satisfies Table 1. In [13], Pinhanez proves that theresult of this algorithm always contains all possiblesolutions of the network (that is, it is conservative),and it is linear in the number of constraints. Pinhanezalso shows that arc-consistency produces a reasonableapproximation of the minimal domain (see Pinhanez[13] for details).

Table 1 Admissible values for primitive temporal relations in aPNF-network

Admissible values of B when A r B

r A = {P} A = {N} A = {F}

equal P N Fbefore PNF F Fi-before P P PNFmeet PN F Fi-meet P P NFoverlap PN NF Fi-overlap P PN NFstart PN N Fi-start P PN Fduring PN N NFi-during P PNF Ffinish P N NFi-finish P NF F

8

4.3 Time expansion

PNF-restriction deals exclusively with determining fea-sible options of an action at a given moment of time.However, information from a previous time step can beused to constrain the occurrence of intervals in the nextinstant. For example, after an interval is determined tobe in the past, it should be impossible for it to assumeanother PNF-value, since, in our semantics, the corre-sponding action is over. To capture this concept, wedefine a function that time-expands a component do-main into another that contains all the possible PNF-values in the next instant of time for each node.

Formally, we define a function Tm, called the timeexpansion function, that considers a component domainW t at time t and computes the component domainW tþ1 ¼ Tm W tð Þ at time t þ 1 by considering what thepossible states are in t þ 1. To define that function, westart by defining a time expansion function sm for eachelement of fpast; now; futureg, such as:

sm pastð Þ ¼ P sm nowð Þ ¼ PN sm futureð Þ ¼ NF

Notice that sm assumes that between two consecutiveinstants of time there is not enough time for an intervalto start and finish. That is, if a node has a value future attime t, it can only move to now, but never straight to thepast state at time t þ 1. Based on sm, we define thefunction that expands the elements of M , Tm : M ! Mas being the union of the results of sm,

Tm Xð Þ ¼[

x2Xsm xð Þ

and the time expansion of a component domain W,Tm : U ! U (abusing the notation), as the component-

wise application of the original Tm on a componentdomain W ¼ Wið Þi,Tm Wð Þ ¼ Tm W1ð Þ;Tm W2ð Þ . . .Tm Wnð Þð Þ:Pinhanez [13] shows that it is possible to define a similartime expansion function that allows an interval to startand finish between two consecutive instants of time. Forsimplicity, we limit the discussion in this paper to thefunction defined above.

4.4 PNF propagation

Now that we have methods both for constraint propa-gation and for incorporating time information, we areready to define the temporal reasoning foundation ofour run-time engine. The goal is to determine, for eachinstant of time, which PNF-states are compatible withthe current information coming from sensors, the pre-vious state of the system, and the constraints betweenthe intervals.

Let us consider an initial component domain W 0 asbeing composed only of PNF, W 0 ¼ PNFð Þi. Then, foreach instant of time t, we can determine through sensorinformation or external sources the PNF-state of someof the nodes. With this information, we create thecomponent domain V t containing all these knownvalues and assigning PNF for the other nodes. Then,given the previous component domain W t�1, we cancompute an upper bound of the current minimaldomain of the PNF-network (see the proof in [13]) bymaking

W t ¼ R Tm W t�1� �\ V t

� �

Fig. 7 Algorithm to computearc-consistency

9

We call this process PNF propagation. Notice that themore information contained in V t, the smaller is W t. Inthe extreme case, if V t is the minimal domain, then W t isalso the minimal domain. In most cases, however, wewill have V t providing new information that is filteredthrough the intersection with the past information(provided by TmðW t�1Þ). Then, PNF-states incompatiblewith the structure of the problem are removed by re-striction, through the computation of the minimal do-main. In practice, we can employ the arc-consistencyalgorithm as defined in Fig. 7 instead of restriction toassure that computation occurs in linear time.

Let us examine an example of PNF propagation.Suppose a simple system where an on/off motion sensorhas to turn off the lights of a room whenever someoneleaves the environment. We can represent this situationby an IA-network composed of two intervals, A and B,where A corresponds to the motion sensor detectingpeople in the environment and B the state where thelights are off. To program the automatic turn off of thelights, we simply state A meet B. The standard way torepresent the on/off states of A in the PNF formalism isto assign the PNF-value N to the PNF-state of A whenA is on, and PF when A is off.

Suppose that the initial state of the sensor A is off(PF) while B’s is unknown (PNF), represented byW 0 ¼ PF ; PNFð Þ. Assume that in the next instant oftime, t ¼ 1, A turns on, represented by V 1 ¼ N ; PNFð Þ.Using PNF propagation,

W 1 ¼ R Tm W 0� �

\ V 1� �

¼ R Tm PF ; PNFð Þð Þ \ N ; PNFð Þð Þ¼ R PNF ; PNFð Þ \ N ; PNFð Þð Þ¼ R N ; PNFð Þð Þ ¼ N ; Fð Þ

The result, W 1 ¼ N ; Fð Þ, yields that since A is on, then Bhas not occurred yet. If in the next instant of time, t ¼ 2,the motion sensor fails to detect a user in the environ-ment, V 2 ¼ PF ; PNFð Þ, we obtain that the lights must beon. Indeed,

W 2 ¼ R Tm W 1� �

\ V 2� �

¼ R Tm N ; Fð Þð Þ \ PF ; PNFð Þð Þ¼ R PN ;NFð Þ \ PF ; PNFð Þð Þ¼ R P ;NFð Þð Þ ¼ P ;Nð Þ

5 The interval scripts run-time engine

Programming in interval scripts is achieved by definingtemporal constraints between nodes associated to sen-sors and actuators of the interactive environment, agent,or computer characters. The constraints describe thedesired temporal structure of the occurrences of theactions and states associated with the sensors and ac-tuators. However, an interval script only declares howthe actions and sensed states should and should nothappen when the interactive environment or agent isrunning. To actually control the environment or agent, itis necessary to use a run-time system that collects the

information from the sensors and, based on the intervalscript, actually triggers and stops the actions of actua-tors — the interval scripts run-time engine. Althoughthere may be different ways to implement a run-timeengine for intervals scripts (for instance, using Andreand Rist’s constraint propagation method [33]), we de-scribe here an implementation based on the concept ofPNF propagation introduced in the previous section.

The basic cycle of our run-time engine is composed offour stages. First the engine computes the states of allsensors. Second, it applies PNF propagation to deter-mine how the system can act so in the next cycle thestates of all sensors and intervals are compatible with theconstraints. Third, the engine selects a particular com-bination of goal states for all actuator nodes. Andfourth, the engine tries to start or stop actions callingappropriate functions, aiming to achieve node states inwhich all constraints are satisfied.

One way to understand our run-time engine is toregard the engine as one player in a two-player game,and assuming the other player to be the environmentand its users. Every play is composed of two ‘‘moves’’,the first performed by the environment and/or its users(sensed/computed by the first stage of the engine). Thesecond ‘‘move’’ is performed by the actions triggered bythe fourth stage of the run-time engine. The second andthird stages can then be seen as a planning process inwhich the system explores the effects of different com-binations of actions, trying to find a set of actions thatresults in states of the nodes that satisfy the constraints.This planning process, whose computation is based onPNF propagation, assumes that the environment re-mains constant except for the system’s actions. We willlater discuss how to overcome this limitation within thisformalism.

5.1 Start, stop, and state functions

To connect an interval script to a real environment oragent, it is necessary to associate nodes of the intervalscript to actual sensors and actuators. In an intervalscript, this is accomplished by associating three types offunctions to a node:

� START: a function that determines what must be doneto start the action associated with a node;� STOP: a function defining what must be done to stopthe action associated with a node;� STATE: a function that returns the actual currentstate of the interval (in our implementation as a dis-junction of the three values, past, now, or future) cor-responding, respectively, to the situation where thenode has already happened, is happening in that mo-ment of the time, or it has not happened yet.

Given the difficulty of controlling actuators in real en-vironments, START or STOP do no need to guaranteethat starting/stopping is achieved immediately, or at all.As we shall see in the rest of this section, the interval

10

scripts run-time engine has ways to cope with actuators’delays and failures. To do this, the STATE functionsshould be oblivious to START and STOP functions andshould be defined so they provide a real assessment ofthe actual situation of the interactive environment andthe on-going actions of the system and its users. STATEfunctions are used both to define nodes attached tosensors and to provide the actual state of execution of anaction. As we see, in interval scripts we de-couple thegoal of starting or stopping an action (represented by theSTART and STOP functions) from the actual state of itsassociated node.

5.2 Choosing globally consistent states for actions

Before we describe the details of the run-time engine, it isnecessary to understand an additional concept. To doso, let us examine a very simple case of a network of twonodes, A and B, constrained to be temporally equal. Forinstance, the two nodes could correspond to the state oftwo devices that are required to be always turned on atthe same time, such as the lights and the ventilation in abathroom control system trying to minimise energyconsumption by turning on/off the devices only whenpeople are inside the bathroom. Let us also assume thatdifferent sensor devices separately control the lights orthe ventilation, but the overall goal is to have the twoactuators always in the same activation state.

Figure 8 shows a situation where the PNF-state ofnode A is PF and B is N, and therefore their values donot satisfy the constraint between them. In that situationthe system should take one of the two actions: try tostart the action corresponding to node A (so its PNF-state becomes N) or to stop node B (so its PNF-statebecomes P). However, as shown in Fig. 8, there are fouroptions of sets of actions to be taken, considering thatthe no action can be done, one of the two, or both. Inparticular, notice that if the system decides both to startA and to stop B, the constraint remains violated.

This example shows that when choosing a set of ac-tions it is better to look for ones whose outcome lead tothe global consistency of the network. If we look onlylocally in each node for a way to make the networkconsistent, it might happen the nodes change their state

but keep the network inconsistent as, in the example ofFig. 8, it is the case when both A is started and B isstopped. However, this example also shows that thereare multiple, often contradictory, ways to satisfy theconstraints of a network.

In our run-time engine, we choose which intervals tostart or to stop by looking for global solutions for theIA-network corresponding to the achievable PNF-states. However, as shown in Fig. 9, running PNFpropagation on the current state generates a stateðPN ; PNÞ which represents not only the two differentpossible solutions, ðN ;NÞ and ðP ; PÞ, achievable bystarting the A action or stopping the B action, respec-tively, but also the PNF-states ðP ;NÞ and ðN ; P Þ whichare not compatible with the constraint equal.

To decide which action to start or stop, we could ex-amine every possible combination of PNF-values con-tained in the result of the PNF propagation and discardthose who are not solutions. This is clearly an exponen-tial search. In practice we have used heuristics that ‘thin’the result of PNF propagation, as described next, al-though always looking for a globally consistent solution.

5.3 Basic cycle of the run-time engine

We now detail how the state of the intervals is deter-mined during run-time and the process that triggers thecall of START and STOP functions. At each time stepthe goal is to select which START/STOP functions tocall in order to make the state of the world in the nextcycle, represented by the PNF-state of the nodes,consistent with the temporal constraints of the PNF-network associated to the script.

Figure 10 shows the diagram of one cycle of our run-time engine. The main component is the current state St

at time t that contains the current state of all intervals inthe associated PNF-network. Each cycle t is composedof the following four stages:

I. Determination of the current state: all the STATEfunctions are called and the engine waits until all theresults are reported back and stored in the componentdomain St. For this computation, the STATE functionis allowed to use the previous state of the intervals,available in the previous component domain St�1, and

Fig. 8 Example of an inconsis-tent state and the effect of possi-ble actions

11

the current states of all the intervals that had alreadybeen computed. After this stage, the current state re-mains unchanged for the rest of the cycle.

II. PNF propagation: in this stage the engine tries to de-termine which actions the system can perform to satisfythe constraints in the next cycle t þ 1, noted as P tþ1,assuming that the state of the environment does notchange. For this, we use a variation of the idea of PNFpropagation where we consider for temporal expansiononly those intervals that can be started or stopped,

V tþ1i ¼

Tm Sti

� �if the interval i has STARTor STOP functions

Sti otherwise

8<

:

By computing PNF propagation only on the nodes thatcan be started or stopped, the system only examineschanges that can happen as a result of its own actions,ignoring all the possible changes in the environment thatcould be detected by the sensors. In practice, the run-time engine behaves as it was in a two-player gameagainst the environment and its users. While in stage I itdetects the ‘play’ of the environment, in this stage itexplores all its own ‘moves’, eventually leading to adecision in stage III, and to action in stage IV.

Using the definition above, this stage then PNF-propagates the current state,

P tþ1 ¼ R Tm Stð Þ \ V tþ1� �¼ R V tþ1� �

If the computation of P tþ1 ¼ RðV tþ1Þ detects a conflict,P tþ1 ¼ [, the engine tries to enlarge the space of pos-sibilities by expanding all states regardless of the exis-tence of START or STOP functions,

P tþ1 ¼ R Tm Stð Þð ÞAlthough this normally handles most problems, thereare situations where sensors report incorrectly andproduce a state with no possible solutions. In those ex-treme cases, the engine simply time-expands the currentstate without computing the restriction,

P tþ1 ¼ Tm Stð Þ

III. Thinning: the result of stage II is normally too bigand undetermined, although it contains all the globallyconsistent solutions. To have a more specific forecastof the next stage without doing search, we apply aheuristic based on the principle that minimal systemaction is desirable: the current state of an intervalshould remain the same unless it contradicts the resultof stage III. This is accomplished by taking a specialintersection operation between the forecast result of theactions in the next state P tþ1 and the current state St.For each node the special intersection is computed by

Fig. 9 Finding a globally consis-tent state for the situationdepicted in Fig. 8

Fig. 10 A cycle of the interval script run-time engine

12

Sti \ P tþ1

i ¼ Sti \ P tþ1

i if Sti \ P tþ1

i 6¼ [

P tþ1i otherwise

(

As seen in the formula, this special intersection opera-tion simply computes a component domain where thePNF-value of each interval is exactly the intersection ofits current value and the possible globally consistentvalues it can assume if action(s) is (are) taken by thesystem. If the intersection is empty, it means that thecurrent state is incompatible with the result of anyglobally consistent action of the system, and therefore,the state of the node has to change as represented inP tþ1

i . If the intersection is non-empty, we have a nodewhose current state (or a fraction of it) is possiblycompatible with the predicted results of possible actions.Following the minimal change heuristics, the thinningprocess makes the goal of the engine to keep the PNF-state of this node i to be the intersection of the currentstate St

i and the forecast P tþ1i .

The component domain that results from the com-ponent-wise intersection is then passed through PNFrestriction to assure that there are solutions and to re-move impossible states, yielding the desired state Dtþ1

Dtþ1 ¼ R St\ P tþ1� �

If the computation of Dtþ1 ¼ R St\ P tþ1� �yields a state

with no solutions we ignore the intersection,

Dtþ1 ¼ P tþ1

In practice ignoring the intersection normally preventsany action to be taken since the states of P tþ1 tend to benot as small as required to call start and stop functionsas described next. Notice that it is not necessary, in thiscase, to apply the PNF restriction to P tþ1, since this wasalready been done in the previous stage and restriction ismaximal.

IV. Taking action: the result of thinning, Dtþ1, is com-pared to the current state St, and START and STOPfunctions are called if the need to change the state ofan interval is detected. To choose among equallypossible alternatives such as in the case of Fig. 8, weuse a heuristics that makes the system more willingto start an action than to stop one. Actions areperformed by the system according to the followingtable:

For example, if the current state of interval i can befuture, F � St

i , and the desired state is now, N ¼ Dtþ1i ,

then the START function of the interval is called.Notice that actions are started both if the desired stateis N or PN, while they can be stopped only if thedesired state is exactly P. In practice, this simple dis-

tinction is sufficient to make more likely the startingthan the stopping of an action.Notice also that the START function is never called ifthe current state of its corresponding interval doesnot contain the future state. That is, an interval isstarted only if there is a possibility that interval is inthe future state. However, a STOP function is some-times called even if the current state of the intervaldoes not contain the now state. In this case we employa mechanism by which if a STOP function is called andthe state is exactly future, then the state of the intervalis automatically set to past, without actually calling itsSTOP function.

To perform the search for globally consistent solutionsinstead of the thinning process of stage III, this stagewould be substituted by the search of all solutions andby a mechanism to arbitrary choose one solution amongthem (for instance, by considering the first solutionfound). As discussed in Pinhanez [13], it is possible touse arc-consistency to speed up considerably suchsearch. In practice, the run-time engine with the minimalchange heuristics described above has been able to runall our applications without halting, deadlocking, or flip-flopping.

Let us examine the behavior of the run-time engine inthe example of Fig. 8 where intervals A and B are con-strained by an equal relation and their initial states arePFand N, respectively. To simplify notation, let us assumethat t ¼ 0 and the current state determined at stage I isS0 ¼ ðPF ;NÞ, where PF corresponds to the state of A andN to the state of B. As shown in Fig. 9, the result of PNFpropagation is P 1 ¼ PN ; PNð Þ. The first attempt of thin-ning the result of PNF propagation yields S0\ P 1 ¼P ;Nð Þ, which is not consistent, D1 ¼ R S0\ P 1

� �¼

R ðP ;NÞð Þ ¼ [. By the recovery heuristics, we define thedesired state to be D1 ¼ ðPN ; PNÞ. According to the stageIV of the run-time engine, only the START function of Awould be called. The choice of starting A over stopping Bis, in this case, an arbitrary by-product of the actiontaking heuristics of stage IV that tends to favor startingactions over stopping them.

5.4 Example of a basic run without conflicts

Figure 11 shows an example of a run of the intervalscript of It described in Sect.3. In the first instant of therun t ¼ 0, we assume that the state of all intervals is F.Because user makes a pose is a sensor, i.e. withoutSTART/STOP functions, its PNF-state F is not ex-panded to NF during the temporal expansion process ofstage II. Although the result of time expansion wouldallow some of the action intervals to be also in the Nstate, the F state of user makes a pose constrains all otherintervals to be in the F state. This is detected by theresult of the PNF propagation process, P 1, which keepsthe state of the system as it is. Given that D1 ¼ P 1, noaction is taken.

x � Sti Dtþ1

i action

F N, PN STARTN P STOPF P STOP

13

In the next instant of time, t ¼ 1, the interval usermakes a pose is detected, as reported by the N valuecoming from its STATE function. When PNF propa-gation is run, the fact that user makes a pose and cam-object takes a picture should start together constrains thedesired PNF-state (D2) of the latter to be N, generating acall for the START function of camobject moves front.Notice also that in this situation, the propagated PNF-state (P 2) of the camobject clicks interval (as well as thestate of camobject moves back) is NF, meaning that theymay either be occurring or may occur in the future.However, they are not triggered because the thinningheuristics prevents change in intervals whose propagatedPNF-state (in this case, NF) contains the current state(F). In the next instant, t ¼ 2, we assume that camobjectmoves front successfully started and therefore no furtheraction is required. The system remains in this state up tot ¼ 12.

When t ¼ 13, camobject moves front has finished andtherefore has the P state. Because of the meet constraint,camobject clicks should start immediately. Notice thatthe result of PNF propagation, P 14, shows exactly thatconfiguration and the appropriate action is taken.In t ¼ 14 the desired state is reached for the interval.

When camobject clicks finishes at t ¼ 20 we have asimilar situation where the START function of camob-ject moves back is called. However, this time we assumethat, for some reason, the camera does not start movingback in the next instant of time, t ¼ 21. As shown inFig. 11, the discrepancy between the current state F (S21)and the desired state N (D22) persists, generating a sec-ond call for the START function of camobject moves

back. At t ¼ 23 the interval starts running and the sit-uation is kept until the interval ends. At t ¼ 31 the Pstate of camobject moves back is detected by the STATEfunction of camobject takes a picture (as described in thenext section), and all the intervals go to the P state.

5.5 Run with conflicts

An example of a run with conflicts is show in Fig. 12 forthe same interval script. Here, we consider the casewhere the intervals camobject takes a picture and user ismoving are defined as mutually exclusive and a conflictsituation arises when both are expected to happen.

At t ¼ 31, a conflict is detected by the PNF propa-gation stage since there is no way to satisfy the re-quirements that camobject takes a picture should starttogether with user makes a pose and that camobject takesa picture can not happen while user is moving is occur-ring. In this situation the first level of recovery succeedsand finds a set of states that is compatible with theconstraints as shown in Fig. 12. Basically, the systemkeeps the current situation as it is and does not startcamobject takes a picture. Later, when t ¼ 37, user ismoving finishes, and since user makes a pose is stillhappening, camobject takes a picture is started.

As we see, the run-time engine of interval scripts wasdesigned to avoid halts by searching for feasible stateswith the least amount of change. This strategy is notguaranteed to succeed but has proved to be robust en-ough to run quite complex structures. We are currentlyworking on better methods to overcome contradictionssuch as keeping a history of previous states for back-tracking purposes (such as those described in Pinjanez

Fig. 11 Example of a run of the interval script described in Sect. 3

14

[13]) and devising mechanisms to detect and isolate thesource of conflicts.

6 The interval scripts language (ISL)

The run-time engine described in the previous sectionhas been implemented and used in the interactive envi-ronments described in Sect.7. For programming pur-poses, we have defined a text-based language called theinterval scripts language, or ISL, that provides a simpleway to describe an interval script. In the this section weshow a programming example of ISL.

Although there are multiple ways to implement theconcepts described in this paper, we chose to develop atext-based language containing the descriptions of theintervals and the temporal constraints between them. Inour implementation, the file containing the languagestatements is run through an ISL compiler that producesa set of C++ files. The C++ files can be compiled usingany standard compiler and executed at run-time usingspecially defined functions defined in another C++ file.It should be noticed that ISL is a particular implemen-tation of the paradigm of interval scripts. As discussedlater in Sect.8, it is possible to program with intervalsand temporal constraints using, for instance, a graphicalinterface.

This section covers just the basic syntax and seman-tics of ISL, using as example the implementation of theinteraction described in Sect.3. More details about thesemantics of the language and its complete grammar canbe found in Pinhanez [13].

6.1 Encapsulating code in intervals

We start our description of ISL by examining how thelanguage communicates with low-level programming

languages that provide basic functions and device access.In ISL, we simply allow the inclusion of C++ code di-rectly into the script.

To begin our example, we show how to connect aninterval in ISL to low-level C++ functions in the contextof the example described in Sect.3. Figure 13 shows thedefinition of the interval camobject moves front in ISLwhere a camera-like CG object referred to as camobjectmoves to the front of the screen. The definition of theinterval is comprised between curly brackets containingthe definition of the interval’s basic functions. To includeC++ code we use the command execute followed by thesymbols ‘[>’ and ‘<]’. For instance, when the STARTfunction is called, it executes the C++code between thosesymbols, that is, a C++method called MoveFront for theobject camobject with parameter value ‘1:0’. The STOPfunction is defined in a similar way.

The definition of the STATE function is slightly dif-ferent. In this case, the function is defined to set the stateof the interval to be equal to the state returned by theC++ code inside the special symbols. In the case de-picted in Fig. 13, a method of the object camobjectdetermines if the computer graphics camera is moving ornot. If true, the state of the interval is set to now, referred

clicks

Fig. 12 Example of a run withconflict

Fig. 13 Interval script describing the forward movement of thecamobject object.

15

in C++ code by the special constant _N_; otherwise, thestate is set to be past OR future, symbolised by P F.

6.2 Describing constraints

Using the example of Sect.3, we examine how con-straints between intervals are defined in ISL. Figure 14shows the script corresponding to the two constraintsimposed over the three intervals camobject moves front,camobject clicks and camobject moves back. The scriptfirst defines the intervals (here omitted for clarity), fol-lowed by two statements establishing the temporal

constraints between the intervals. Notice the syntaxbetter-if that was chosen to imply that this is a constraintthat the system will try to enforce but that is not guar-anteed to occur.

6.3 Defining on previous intervals

A key feature of ISL is the possibility of defining a newaction or world state solely based on other intervals.With this we can create hierarchies, abstract concepts,and develop complex, high-level scripts. Continuing withthe example of Sect.3, we define the interaction betweenthe user and the camera-like object as follows: when theuser makes a pose (as detected by a computer visionsystem), the camera takes a picture, an action that iscomposed of moving the camera forward, closer to theuser, clicking, and returning to its original position.Figure 15 shows the program corresponding to this in-teraction.

Initially the interval user makes a pose is defined asbefore, by a reference to C++ code that communicateswith the vision system. Notice that this interval corre-sponds to a sensor and therefore has no STARTor STOP functions. To accomplish the clicking ofthe camera and the taking of the picture, the interval

Fig. 14 Interval script describing three intervals and two constraintsamong them

Fig. 15 Scripting based onpreviously defined intervals

16

camobject takes a picture is defined, based on its threecomponent intervals. First, a STOP function is definedso if the interval is happening and it is made to stop, itfirst makes the camera-like object move back to itsoriginal position before actually stopping the main in-terval. Notice that, as showed in this example, stoppingan interval may require actions to be executed and,therefore, some time is required before an interval ac-tually stops. If the STOP function was not explicitlydefined, the interval still could be stopped but in thatcase the interval camobject moves back would not nec-essarily be executed.

This is also the case of the STATE function thatdefines the computation of the current state of the in-terval camobject takes a picture. As shown in Fig. 15, ifany of the component actions is happening, the state ofthe interval is set to now. If the action of moving backthe camera has occurred, that is, its current state is past,then the interval camobject takes a picture is determinedto have already happened and its own state is set to past.Notice that, according to this definition, when the STOPfunction of the interval is called, the interval is notconsidered finished until the object completely movesback to its original position.

Finally, following that example, we want thatwhenever user makes a pose starts happening, thecamera takes the picture. This is established bythe constraint start OR equal OR i-start that forcesthe two intervals to start together. Also, we imposeconstraints that relate the interval camobject takes apicture to its three constituents. Notice that, it is notnecessary to explicitly define a START function forcamobject takes a picture, since the constraint startbetween the intervals camobject moves front and cam-object takes a picture establishes that the triggering ofeach interval immediately starts the other. And, com-pleting the example of Sect.3, the interval user ismoving is defined and constraints are imposed thatmake it mutually exclusive to the interval camobjecttakes a picture.

The Interval Scripts Language includes many otherfeatures that have shown to be extremely helpful in theprogramming of the interactive environments describedin Sect.7. See Pinhanez [13] for a detailed description ofthose features.

7 Working with interval scripts

Evaluating a programming method is always difficult.Although we have not performed formal studies on howeasy it is for programmers to understand and use theparadigm and the ISL language, we describe three sys-tems here built using interval scripts as evidence towardsour belief that the language is an improvement overcurrent paradigms and programming methods for in-teractive environments and agents. We do not see howthese systems (especially the last two) could have beenprogrammed using event-loops or state-machines in a

reasonable amount of time and without major problemsin debugging and maintenance. In fact, all the controlstructures of the systems described here were developedin very short periods of time. Based on these experiences,we believe that our main objectives when designing in-tervals scripts were achieved, that is, that the paradigmprovides expressiveness and simplicity beyond currentprogramming methods.

The first two cases of interactive environmentsdeveloped with interval scripts, SingSong and It/I, areinteractive theatrical performances where human andautonomous computer-graphics actors interact follow-ing a story. They constitute what is called computertheater, a term referring to live theatrical performancesinvolving the active use of computers in the artisticprocess (see Pinhanez [43]). The third system, It, is aninteractive art installation where interval scripts wereused not only to program the environment but also itsinhabitant computer characters and agents. Although allthree interactive environments correspond to entertain-ment/art applications, it is reasonable to expect thatsimilar development and performance results would beobtained in other scenarios of interactive environmentssuch as computer-controlled offices or homes. In fact,given the complexity of everyday life, we expect that apowerful programming paradigm such as the intervalscripts is even more necessary in practical cases toachieve similar behaviour complexity. For instance, oneof the authors of this paper, Claudio Pinhanez, hasparticipated in the design and development of a proto-type of a computer-augmented office environment (seeLai et al. [5] for a description of the system). In thatproject, it was observed that the lack of a more complexprogramming model constrained the diversity andadaptability of the behaviours of the space to simplereactive mechanisms.

7.1 SingSong: an interactive performance environment

SingSong was our first experience with interval scripts[17]. SingSong is composed of a large video screen thatdisplays four computer graphics-animated charactersthat can ‘‘sing’’ musical notes (as produced by a MIDIsynthesizer). A camera watches the user or performerdetermining the position of his/her head, hands, andfeet. The body position is recovered by the softwarepfinder developed at the MIT Media Laboratory [44].

All the interaction is physical, non-verbal: the user orperformer gestures and the CG-characters sing notesand move. SingSong was produced and performed in thesummer of 1996. Although it is a short play of aboutfour minutes, the interval script involves around 200intervals, including a great deal of low-level control ofI/O devices. It took about two days to write and debugthe controlling program. Figure 16 shows a sequence ofscenes from SingSong.

SingSong used the first version of the concept of in-terval scripts, described in Pinhanez et al. [17]. The

17

SingSong version of interval scripts already included theimportant idea of separating the desired action fromthe actual action. However, in that version, the idea wasimplemented using two implicitly defined intervals in-stead the simpler use of START, STOP, and STATEfunctions (see Pinhanez et al. [17]). The use of twointervals required a more complicated model of the sit-uation, making scripting more difficult than in the sub-sequent versions.

The main problem with the SingSong version of theinterval scripts idea was that the script had to be writtenexplicitly into C++ code. Although the interval scriptsparadigm helped to avoid extremely complex and un-debuggable control structures, after a while it becamequite difficult to read the C++ file. This realisationprompted us for the development of the Interval ScriptsLanguage introduced in the previous section.

7.2 It/I: a computer theatre play

Following SingSong we decided that it was necessary todevelop a more comprehensive test for interval scripts.Particularly, we were interested in testing the robustnessof the interval scripts paradigm in conditions where

Fig. 16 Scenes from SingSong

Fig. 17 Scenes from It/I. The computer graphics objects on the screenare autonomously controlled by the computer character It

18

difficulties in scripting or run-time failures would becritical.

With these ideas in mind, one of the authors of thispaper, Claudio Pinhanez, wrote the computer theaterplay It/I. The play is a pantomime where one of thecharacters, It, has a non-human body composed ofCG-objects projected on screens (see Fig. 17). The ob-jects are used to play with the human character, I. It‘‘speaks’’ through images and videos projected on thescreens, through sound played on stage speakers, andthrough the stage lights.

The play is composed of four scenes, each being avariation of a basic cycle where I is lured by It, is playedwith, gets frustrated, quits, and is punished for quitting(see Pinhanez and Bobick [14] for a detailed script of theplay). It is important to note that any individual scene ofIt/I has a complexity level beyond previous full-bodyinteractive systems [35, 45]. The play was produced inthe summer/fall of 1997 and performed six times for atotal audience of about 500 people. Each performance,lasting 40 minutes, was followed by an explanation ofthe workings of the computer-actor. After that membersof the audience were invited to go up on stage and playthe second scene from the play (see Pinhanez and Bobick[14]).

For performance safety, we implemented each sceneof It/I in a separate script, generating four differentexecutable files that were loaded in the beginning of thecorresponding scene. Table 2 summarizes the composi-tion of the interval scripts of each scene of It/I,displaying the total number of intervals (i.e. the numberof nodes of the PNF-network) and the number of con-straints. As can be seen in the table, we have more than100 nodes per scene with an average of 80 constraintsbetween them.

Although simple, the design of the run-time engine ofthe It/I version of interval scripts was not able to takefull advantage of the prediction powers of constraintpropagation of the current version of the run-time en-gine, where the idea of computing the desired state byPNF propagation and thinning (as described in Sect.5)considerably improved the robustness of the system.

7.3 It: an Interactive interactive installation

After It/I, we developed an interactive art installationcalled It where users can experience the story of the play

without having previously watched it. In It, the userenters the area, is approached by the character It, startsplaying with the CG-objects as I did in the original play,and like I, is not allowed to leave the space or stopplaying. The installation was finished in the spring of1999 and demonstrated successfully during two openhouses held at the MIT Media Laboratory.

In It we employed the version of ISL introduced inSect.6, and the run-time engine discussed in Sect.5. TheISL was used both to program the environment and itsstory and the computer agents that control the charac-ters and agents that inhabit it. In fact, the controlstructure of It is distributed among four agents: the storyagent, that controls the basic development of the envi-ronment; the It character and the I character agents,corresponding to the two main characters of the inter-active story; and the light designer agent, that controlsall the lighting effects of the environment.

In Table 3 we summarise the composition of the in-terval scripts used for the different agents of It. Table 3shows that interval scripts can be used to construct andcontrol large and complex collections of actions. Forexample, the It character agent employs 224 nodes for atotal of more than 350 constraints. During run-time, weapply PNF propagation on this network at the rate of 20Hz without any visible impact on the performance of theSGI R10000 machine used in the installation. In fact, allfour agents run simultaneously in one single machine,consuming at most 15% of the CPU time, including thecommunication overhead.

8 Future directions

Although the interval scripts language has been fullyimplemented and intensively tested in three differentprojects, there is still work to be done in many direc-tions. First, it would be interesting to get the system to astate where it could be released to other researchers andprogrammers interested in using interval scripts to buildinteractive systems. In this context it becomes possible toevaluate the difficulty in learning the language and whichfeatures different types of programmers would like tohave added to the language.

In particular, an area we want to investigate is thebest interface for the interval scripts programming par-adigm: text-based or graphical. In the case of a graphicalinterface, a possible approach is to allow the user to

Table 2 The composition of the interval scripts for the differentscenes of It/I

Total numberof nodes

Total numberof constraints*

scene I 120 88scene II 80 70scene III 92 66scene IV 115 97

(*) including implicit constraints (see [13]).

Table 3 The composition of the interval scripts for the differentagents in It

No. of nodes Total no.of constraints

story 59 101It character 224 355I character 8 8light designer 94 133

19

construct temporal diagrams similar to Fig. 4. However,since we allow disjunction of temporal constraints, thereare always multiple possibilities for the visual arrange-ment of the intervals and therefore the visualisation maycreate more confusion than to provide help. Neverthe-less, we have always found very useful to draw temporaldiagrams when programming in ISL.

We would also like to integrate into the intervalscripts paradigm mechanisms that represent actions informs that the machine can reason about. Currently, therun-time engine can only control the execution of scriptsbut has no mechanisms to create action plans beyondthe simple 1-step look-ahead performed in stage II.Comparing, for instance, to the Hap system developedby Loyall and Bates [46], the current interval scriptscontrol system is basically reactive. Although it can runand control interactive plans with very complicatedtemporal structures, it has no way to construct plansfrom goals beyond the immediate one-step future. Thislimitation is partially due to the ideas employed to re-duce the complexity of temporal constraint propagationas described in Sect.4.

We have considered different mechanisms to over-come this limitation. A basic idea is to perform multi-step look-ahead, by considering all possible differentcombinations of sensor input. In Pinhanez [13] we showthat it is possible to maintain multiple threads of sensorvalues, under some simplifying assumptions, because amajority of the generated threads become inconsistentand can be discarded after some cycles, due to con-tradiction with real sensor values. Since that work wasfocused on action recognition, this was done in thecontext of keeping multiple possibilities for the events inthe past. Similarly, we could use multiple threads ofsensor values to search the future for an activationpattern that achieves a determined goal expressed as acombination of PNF-states for the different intervals ofa script. To accomplish this, it is necessary to incorpo-rate mechanisms that forecast the likely duration ofactions and the likely delays in starting and stoppingthem, so the systematic PNF-style exploration of thefuture becomes feasible.

9 Conclusion

In this paper, we propose the interval scripts paradigmfor programming interactive environments that is basedon declarative programming of temporal constraints.Unlike previous constraint-based systems, we allow theuse a strong temporal algebra with mutually exclusiveintervals. Among other things, we have created a para-digm that adequately handles the representation andcontrol of some of the main issues in the programmingof actual interactive environments: delays, continuity,and history of actions and states; mutually exclusivelyactions; stopping of long actions; and recovery fromconflicts and contradictions. Through examples we haveshown that our paradigm allows good expressiveness

and facilitates the management of context in an inter-active environment.

This paper significantly extends the initial proposal ofinterval scripts outlined in Pinhanez et al. [17]. Amongother innovations, the run-time engine described in thispaper is able to actively forecast future states that satisfythe temporal constraints, allowing the search for a con-sistent set of action triggerings. Also, the definition of theInterval Scripts Language greatly simplifies the use oftemporal constraints in the programming process.

From the experience acquired in the use of the par-adigm for the implementation of three different projectsin the entertainment/art domain, we believe that pro-gramming systems incorporating the interval scriptsparadigm can significantly ease the design and buildingof interactive systems in the future.

Acknowledgements SingSong’’ was written, directed, and per-formed by Claudio Pinhanez. It/I was written and directed byClaudio Pinhanez, and produced by Aaron Bobick; the crew wascomposed by John Liu, Chris Bentzel, Raquel Coelho, LeslieBondaryk, Freedom Baird, Richard Marcus, Monica Pinhanez,Nathalie van Bockstaele, Maria Redin, Alicia Volpicelli, NickFeamster, and the actor Joshua Pritchard. It was designed andimplemented by Claudio Pinhanez based on the elements producedfor It/I.

Claudio Pinhanez was supported in different stages of this re-search by the scholarship from CNPq, process number 20.3117/89.1; by the DARPA contract DAAL01-97-K-0103; by the MITMedia Laboratory; and by the MIT Japan Program through theStarr Foundation. SingSong was sponsored by the Media Inte-gration and Communication (MIC) laboratory of the AdvancedTechnology Research (ATR) laboratories in Kyoto, Japan. It/I andIt were sponsored by the Digital Life Consortium of the MITMedia Laboratory.

References

1. Pausch R, Snoddy J, Taylor R, Watson S, Haseltine E. Dis-ney’s Alladin: first steps toward storytelling in virtual reality.In: Proceedings of SIGGRAPH’96. 1996, pp 193–203

2. Bobick A, Intille S, Davis J, et al. The KidsRoom: a percep-tually-based interactive immersive story environment.Presence: Teleoperators and Virtual Environments 1999; 8(4):367–391

3. Tosa N. Expression of emotion, unconsciouness with art andtechnology. In: Hatano G, Okada N, Tanabe H (eds) Affectiveminds, Elsevier, 2000, pp 183–205

4. Pinhanez C, Davis J, Intille S, et al. Physically interactive storyenvironments. IBM Systems Journal 2000; 39(3&4): 438–455

5. Lai J, Levas A, Chou P, Pinhanez C, Viveros M. BlueSpace:Personalizing workspace through awareness and adaptability.International Journal of Human Computer Studies 2002; 57(5):415–428

6. Crowley JL, Coutaz J, Berard F. Things that see. Communi-cations of the ACM 2000; 43(3): 54–64

7. Raskar R, Welch G, Cutts M, Lake A, Stesin L. The office ofthe future: a unified approach to image-based modeling andspatially immersive displays. In: Proceedings SIGGRAPH’98.Orlando, FL, 1998, pp 179–188

8. Abowd GD. Classroom 2000: an experiment with the instru-mentation of a living educational environment. IBM SystemsJournal 1999; 38(4): 508–530

9. Mynatt ED, Essa I, Rogers W. Increasing the opportunities foraging in place. In: Proceedings ACM conference on universalusability (CUU’00), Arlington, VA, 2000

20

10. Pentland A. Smart rooms. Scientific American 1996; 274(4):68–76

11. Shaw C, Green M, Liang J, Sun Y. Decoupled simulation invrtual reality with the MR Toolkit. ACM Transactions onInformation Systems 1993; 11(3): 287–317

12. Pausch R, Burnette T, Capeheart AC, et al. A brief architec-tural overview of Alice, a rapid prototyping system for virtualreality. IEEE Computer Graphics and Applications 1995

13. Pinhanez CS. Representation and recognition of action in in-teractive spaces. PhD Thesis, Media Arts and Sciences Pro-gram, MIT, Cambridge, MA, 1999

14. Pinhanez CS, Bobick AF. It/I: A theater play featuring anautonomous computer character. Presence: Teleoperators andVirtual Environments 2002; 11(5): 536–548

15. Allen JF. Maintaining knowledge about temporal intervals.Communications of the ACM 1983; 26(11): 832–843

16. Vilain M, Kautz H. Constraint propagation algorithms fortemporal reasoning. In: Proceedings of AAAI’86, Philadelphia,PA, 1986, pp 377–382

17. Pinhanez CS, Mase K, Bobick AF. Interval scripts: a designparadigm for story-based interactive systems. In: Proceedingsof CHI’97, Atlanta, GA, 1997, pp 287–294

18. Coen MH. Building brains for rooms: designing distributedsoftware agents. In: Proceedings of IAAI’97, Providence, CT,1997, pp 971–977

19. Director’s User Manual, MacroMind Inc. 199020. MAX Getting Started Manual, Opcode, 200221. Roads C. The computer music tutorial. MIT Press, Cambridge,

MA, 1996, pp 969–101622. Rowe R. Interactive music systems. MIT Press, Cambridge,

MA, 199323. Rollings A, Morris D. Game architecture and design. Coriolis

Group, Scottsdale, AZ, 200024. Buchanan MC, Zellweger PT. Automatic temporal layout

mechanisms. In: Proceedings of ACM Multimedia’93, Ana-heim, CA, 1993, pp 341–350

25. Hamakawa R, Rekimoto J. Object composition and playbackmodels for handling multimedia data. In: Proceedings of ACMmultimedia’93,Ahaheim, CA, 1993, pp 273–281

26. vanRossumG, Jansen J,MullenderK,BultermanD.CMIFed: apresentation environment for portable hypermedia documents.In: Proceedings of ACM multimedia’93, Anaheim, CA, 1993

27. Vazirgiannis M, Kostalas I, Sellis T. Specifying and authoringmultimedia scenarios. IEEE Multimedia 1999; 6(3): 24–37

28. Bailey B, Konstan JA, Cooley R, Dejong M. Nsync – a toolkitfor building interactive multimedia presentations. In: Pro-ceedings of ACMmultimedia’98 Bristol, UK, 1998, pp 257–266

29. Jourdan M, Layaida N, Roisin C, Sabry-Ismail L, Tardif L.Madeus, an authoring environment for interactive multimediadocuments. In: Proceedings of ACM multimedia’98 Bristol,UK, 1998, pp 267–272

30. Selcuk KC, Prabhakaran B, Subrahmanian VS. CHIMP: aframework for supporting distributed multimedia documentauthoring and presentation. In: Proceedings of ACM multi-media’96, Boston, MA, 1996, pp 329–339

31. Agamapolis S, Bove Jr. VM. Multilevel scripting for responsivemultimedia. IEEE Multimedia 1997; 4(4): 40–50

32. Wirag S. Modeling of adaptable multimedia documents. In:Proceedings of European workshop on interactive distributedmultimedia systems and telecommunications services, Darms-tadt, Germany, 1997

33. Andre E, Rist T. Coping with temporal constraints in multi-media presentation planning. In: Proceedings of AAAI’96,Portland, OR, 1996, pp 142–147

34. Kautz HA, Ladkin PB. Integrating metric and qualitativetemporal reasoning. In: Proceedings of AAAI’91, Anaheim,CA, 1991, pp. 241–246

35. Krueger MW. Artificial reality II. Addison-Wesley 199036. Bederson BB, Druin A. Computer augmented environments:

new places to learn, work and play. Ablex, Norwood, NJ,1995

37. Dey AK, Salber D, Abowd GD. A conceptual framework anda toolkit for supporting the rapid prototyping of context-awareapplications. Human-Computer Interaction Journal 2001;16(1)

38. Pinhanez CS, Bobick AF. Human action detection using PNFpropagation of temporal constraints. In: Proceedings ofCVPR’98, Santa Barbara, CA, 1998, pp 898–904

39. Meiri I. Combining qualitative and quantitative constraints intemporal reasoning. In: Proceedings of AAAI’91, Anaheim,CA, 1991, pp 260–267

40. Pinhanez CS, Bobick AF. PNF propagation and the detectionof actions described by temporal intervals. In: Proceedings ofDARPA image understanding workshop, New Orleans, LA,1997, pp 227–233

41. Dechter R. From local to global consistency. Artificial Intelli-gence 1992; 55(1): 87–107

42. Mackworth AK. Consistency in networks of relations. Artifi-cial Intelligence 1977; 8(1): 99–118

43. Pinhanez CS. Computer theater. In: Proceedings of eighth in-ternational symposium on electronic arts (ISEA’97), Chicago,IL, 1997

44. Wren C, Azarbayejani A, Darrell T, Pentland A. Pfinder: Real-time tracking of the human body. IEEE Transactions PatternAnalysis and Machine Intelligence 1997; 19(7): 780–785

45. Maes P, Darrell T, Blumberg B, Pentland A. The ALIVEsystem: full-body interaction with autonomous agents. In:Proceedings of computer animation’95, Geneva, Switzerland,1995

46. Loyall AB, Bates J. Hap: A reactive, adaptive architecture foragents. Carnegie Mellon University, Technical Report CMU-CS-91-147, Pittsburgh, PA, 1991

21