Abstract — Modern technology offers the possibility to construct

networked monitoring systems from autonomous computing

nodes equipped with appropriate sensors. Complementing such a

system with actuators yields a cyber-physical system that must be

able to cope with uncertainties arising from feedback loops via

the physical world. The common property of such systems lies in

the high degree of uncertainty in the varying configuration of the

system and also in the potentially high amounts of (unstructured)

data that can be generated by such a system. In order to tackle

these problems and make the distributed monitoring systems

more usable the concepts of situational information and

hierarchies of situations can be applied in this domain. The

problem of high amounts of data can be partially solved by

arriving at a higher level of abstraction lower in the processing

chain, communicating only data fused into an ontological

structure to information consumers. The fusion templates are

called situation parameters and values of the fused data items are

called situational parameter values in the context of the current

article. Situation parameter values must be tagged with

situational and temporal validity information in order to cope

with the delays and spatial uncertainties that can occur in a

distributed monitoring system. The lower level situation

parameters can be fused to even higher level situation

parameters, projecting the temporal and spatial validity

information from the lower level parameters up to the higher

level parameters. The article presents the concepts of forming

situational information templates and hierarchies based on data

available from a distributed monitoring system where the

temporal and spatial properties of situational information are

taken into account. A case study is presented that shows the

feasibility of the concepts in a real world monitoring scenario.

Index Terms — cooperative distributed systems, interactive

computing, middleware (for subscription and distribution of

situational information), situation awareness, validation and

verification

I. INTRODUCTION

Networked systems and systems of systems embedded into

the environment are gaining importance in the modern world.

The concept of Network Enabled Capabilities, but also

solutions for production management and business

intelligence, require collecting and organizing potentially vast

amounts of information from various sources. In case of all

distributed systems that cater for the improvement of situation

awareness of users or system components the temporal and

spatial aspects of data are of great importance. While in many

cases the consideration of temporal and spatial aspects of data

have been implicitly embedded into the system design,

oftentimes the importance of these aspects has also been

neglected. Considering these aspects explicitly will result in

more predictable and correct performance of the systems.

Using a modified definition of a situation introduced in

sociology [3] we can say that a situation is the aggregate of

biological, psychological, socio-cultural, and environmental

factors acting on an individual or a group of agents to

condition their behavioral patterns. An agent in the current

context denotes a natural (e.g. humans) or an artificial (e.g.

computing systems, or software-intensive agents) agent, and

the environment denotes natural or artificial environment.

Extending the existing situation awareness concepts and

supplementing situational information with temporal and

spatial validity information as suggested in the current paper

enables the creation of systems that employ hierarchical

buildup of situational information.

II. SITUATION HIERARCHIES

Situation management [14] has been mostly viewed from

the perspective of human requirements and processing

capabilities. Computers have over time come to aid humans in

achieving situation awareness. However it is not beneficial to

use the human situational concepts when building up situation

hierarchies computed and managed by a distributed system.

Instead concepts arising naturally from the lowest level of the

processing chain and from fusing these low-level concepts

should be used.

A hierarchical build-up of situation parameter values can be

applied where the lower level parameter values are more or

less directly derived from sensor data. A situation parameter

reflects a property of a parameter of interest, composing them

allows to computing the values of higher level parameters of

interest by using the values of lower level parameters and all

the other relevant information.

In case of an artificial agent the evaluation of situational

parameters can be performed if suitable algorithms exist and

can be executed. In order to guarantee the validity of the

situation assessment, the constraints are specified on the

source data that guarantee the coherence and validity of source

data. In order to deliver the data that satisfies the specified

constraints to the situation parameter computation algorithm,

the concept of mediated interaction [12] must be used. The

mediator can be compared to the channel function in the Q

model [13], as it transmits only data that satisfies the

constraints set by the consumer (i.e. the situation evaluation

algorithm) to the consumer (the situation evaluation

algorithm). The concepts of the validity of situation

parameters and the mediator are elaborated further below.

We consider a situation and hierarchy of situations in the

SITUATION AWARENESS FOR NETWORKED SYSTEMS

Jurgo Preden, Leo Motus, Merik Meriste*, Andri Riid

Research Lab for Proactive Technologies, Dept. Computer Control,

Tallinn University of Technology

*also with University of Tartu, Institute of Technology

Estonia

2011 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support(CogSIMA), Miami Beach, FL

978-1-61284-786-3/11/$26.00 ©2011 IEEE 123

.

context of a networked pervasive computing system that is

also accessible to a number of involved humans. The basic

situations are introduced at the lowest level (i.e. at the level of

embedded computers, and pre-specified humans) in the form

of situation parameters.

Embedded nodes of networked pervasive computing

systems may possess too limited computing power and are

therefore capable of performing simple sensor reading and

data fusion tasks. The basic situations should be defined in

accordance with the capabilities of network nodes and

information processing capabilities of individual humans.

Hence, the parameter values characterizing the basic

situations should stem from the direct readings of sensors,

direct observations or deductions made by humans, or from

elementary data fusion operations. The set of basic situations

(i.e. their parameters) forms the basis for constructing more

abstract situations whose validity depends on the validity of

basic situations. The concept of hierarchical buildup of

situations inspired by Endsley [4] is visualized on Figure 1,

showing the abstraction level of the information elevated in

the chain of situation parameters.

This article presents the concept of situational information

supplemented by validity information, situation hierarchies

and also two case studies, one of situational information

fusion and the other of utilization of situational information by

mobile autonomous agents.

A. Specifying situation hierarchies

When imposing a suitable hierarchical structure to the

process of specifying situations one can either use a top-down

or a bottom-up approach, correspondingly either starting from

the high-level situations and decomposing them to lower-level

situations or starting from low-level situations and looking for

ways to fuse the low-level situations to higher-level situations.

In any case the (automatically) measurable factors for basic

situations must be specified from which more sophisticated

situations can be constructed, measured and reasoned about,

and then used to build up situational awareness for the end

users (which may be natural or artificial agents). It seems to be

most beneficial to tackle the problem from two ends at the

same time – moving from low-level situations to higher level

situations by selecting appropriate fusing algorithms while

keeping the end objective (specific high-level situations) in

mind.

B. Validity of situation parameters

A critical issue of situation awareness is the validity of

situations – when a situation parameter value is computed or

when a situation is identified, it must be possible to estimate

where and when that situation parameter is valid. This

estimation must be performed based on the validity of the

source data – starting from the raw sensor data a situation

inferred from this sensor data is only valid in the region where

the sensor data is valid and also for the period of time when

the sensor data is valid. So the properties of the situational

parameters (regardless of whether these are parameters of the

physical world or the virtual) that are monitored determine the

validity of the situations inferred based on these parameter

values. An example of this is weather monitoring – for most

regions we are able to predict the dynamics of temperature

change and the area where the temperature is homogeneous

quite well, so based on temperature measured in one spot at

one specific moment in time we can quite well estimate the

temperature for the adjacent regions for some period of time.

The data acquired from each sensor (or other source of

information) forms a data stream, from which the situation

parameter values are derived. The temporal (plus spatial and

other characteristic) properties of each stream element are

unique and determined by the phenomena being measured

and/or monitored, by the requirements and/or constraints on

the situation parameter, and by the constraints set by higher

level situation parameter synthesis algorithms that use the

given situation parameter as an input. Any algorithm

computing parameter values for higher-level situation may use

one or more data streams.

It is intuitively obvious that for each situation its parameter

values are valid only for a certain time interval, and/or in a

specific location. Hence the temporal intervals, spatial areas,

and may be some other attributes must be specified for cross-

checking the integrity of parameter values, and thus assess the

validity of obtained situational information. The temporal

validity interval of the situation parameter specifies the time

for which the situation parameter value is valid – the validity

interval depends on the time when the situation parameter

assessment was made and the known dynamics of the

parameter (the property which state that the parameter

expresses). The spatial validity interval specifies the area

where the situation parameter is valid, which naturally

depends on the spatial origin of the source data used for

deriving the situation parameter value and the properties of the

phenomena that the parameter characterizes. For example a

temperature assessment of warm for a room can be typically

considered to be valid only for that room not for example the

back yard which can be viewed from the window of that room

– the validity area of the temperature assessment may ends at

the wall of a room.

The measured values and the estimates stemming from

observations originate from different network nodes, different

sensors, and from variety of persons and need to be checked

for consistency and validity. This can be done automatically

Figure 1 Hierarchical buildup of situation parameters

124

.

only if the measurements and observations are equipped with

attributes that foster the on-line validation procedure.

Every situation parameter value has temporal and spatial

validity values associated with it. The validity values depend

on various aspects, for example the validity area depends on

the location of the agent that acquires the data and on the

properties of the phenomenon being observed, while the

temporal validity interval depends both on the properties of

the environment where the agent is located and on the

phenomenon being observed. The consumer of the situation

parameter values verifies that the validity values of the

parameters do match the constraints set on the incoming data.

The output of the situation assessment algorithm is the

situation parameter value accompanied with the metadata.

C. Formalizing situation parameters

Situation parameter is a three-tuple S ={Sp, St, Sa} where Sp

is the situation parameter value, St is the situation property’s

validity period and Sa is the situation property’s validity area.

The situation parameter value reflects the state of a virtual or

physical property of interest while the validity period and

validity area are metadata on the situation parameter

assessment – what is the period for when and what is the area

for which the situation parameter value holds. For some

situation parameters there is no metadata while for others there

may be more metadata than just the location and time.

A situation parameter value is computed from other

situation parameters or sensor data, both of which must be

associated with metadata Sa =f (S1, S2,…, Sn) . The situation

parameter values used for computing a higher-level situation

parameter value may arrive at different times and at different

rates. A specification of an algorithm that computes the

situation parameter value must contain the type of input data –

the incoming situation parameters. If a sensor reading is used

to evaluate the situation it is expressed as follows: Sb =f (s1)

which should be interpreted that the reading of sensor s1 is

used to compute the value for situation parameter Sb.

D. Situation awareness of a team

In many practical systems a loosely interacting team of

humans and computers has to manage situations composed

from a variety of spatio-temporally distributed natural and

artificial components. Each entity in such a system can be

viewed as an agent on a certain level of abstraction. Situation

awareness of an agent, in the current context, comprises the

confidence in the available situational information and the

ability to use that information in the decision-making process

of the agent. In a realistic system it can be assumed that global

situations are defined for the whole system whereas each agent

may have derived its own situations, departing from its own

goals.

The existence of several types and levels of situations may

be confusing, unless they are (at least partially) harmonized

and prioritized. This leads us to the notion of “team situation

awareness” (also called “distributed situation awareness”);

see, for instance [2]. Formation of team situation awareness

needs specific tools, especially when dealing with a mixed

team of humans and machines. The complexity here lies also

in the fact that situation awareness can be viewed as an

emergent property of collaborative systems, a phenomenon

that resides in the interaction between elements of the system

and not in the individual components of the system [15]. The

first step is to harmonize models of “individual situation

awareness” of team members, followed by harmonization of

individual decision-making procedures. Typically this process

requires negotiations between decisive team members and can

be remarkably accelerated by a suitable negotiation medium,

such as a specially designed middleware for communication.

However, in some cases each agent (component of a

system) may define its own (additional) situations of interest.

This leads us to the notion of “team situation awareness” (also

called “distributed situation awareness”); see for instance [2].

Formation of team situation awareness needs specific tools,

especially if we have a mixed team of humans and machines.

The first step is to harmonize models of “individual situation

awareness” of team members, followed by harmonization of

individual decision-making procedures. Typically this process

requires negotiations between decisive team members and can

be remarkably accelerated by a suitable negotiation medium,

such as a specially designed middleware for communication.

The above definition matches with seminal interpretation,

given by Endsley [5], of situation awareness as a product

resulting from a process of acquiring situational information

and its assessment.

III. MIDDLEWARE FOR SITUATION AWARENESS

In [6] we proposed the concept of middleware as an

active mediator of situational information. In order to make

better use of the situational information (i.e. situation

parameter values) collected by agents in a multi-agent system

the situation parameter values need to be exchanged between

agents. Two aspects make the exchange of such information

complex. Firstly the amount of situational information

available in a complex system containing many agents is

potentially huge. Secondly the configuration of the multi-

agent system is not known before the system starts operating

and the configuration may change at runtime unexpectedly.

The latter means that the sources and consumers of situational

information are established at runtime, making special

requirements on the control of the interaction between the

agents. Since a specific agent requires information that

satisfies the constraints set by that agent there needs to be a

mechanism for locating the required information and

providing it to the agent in a deterministic way.

The term middleware is usually used for a specific software

layer in computing system that connects applications, or

software components and provides functionality that allows

multiple processes that run on one or more system nodes to

interact across a network. We view the middleware as an

active “mediator” of situational information that provides

services for forming team situation awareness. Here the team

is formed by agents who can be humans, computers, smart

sensors, and/or software-intensive devices.

125

.

A. Concept of the middleware as active mediator

The middleware is the mediator that supports formation of

the situation awareness and especially team situation

awareness within a team of autonomous interacting agents.

Conceptually, the middleware is a smart communication

environment that provides services for filtering, partial

validation, and distribution of information according to

personal access rights of agents, and in preferable message

formats for individual agents.

A mediator is required already in case of a single agent

which collects its situational data from a set of sensors (that

can be interfaced directly to the agent) for determining

situation parameter values. It is preferable that the situational

data is tagged with validity information, which enabled cross-

checking of data and validity assessment of its value. The

middleware can validate the consistency and integrity before

the data reaches the agent – this simplifies the substitution of

the agent, or modification of its algorithms. Sensor readings

can be tagged with validity information within a sensor or at

the entry point to the mediator. The validity checks are

performed by the mediator based on the constraints and

requirements provided by the agent that subscribes to the data.

In the case of distributed peers-agents the mediation problem

is more complex since the mediator operates on multiple

simultaneous streams of computation so as each of the

interacting autonomous agents is trying to form its individual

situation awareness, and after that harmonize it with the team

situation awareness.

B. Architecture of the middleware

The middleware is realized by a collection of services at

each computing node. The services are interfaced to an active

mediator that implements the middleware features at each

node. Thus the middleware is formed by the information

mediators and services at individual nodes that interact with

each other. A service-oriented architecture for the middleware

enables easy, self-adjusting communication between dynamic

collections of interacting autonomous agents – those

interactions are required to form individual situation

awareness of agents and to develop the team situation

awareness. The interactions may be divided into multiple

groups depending on tasks and goals of interacting partners –

for instance, the middleware caters for communication

required for sensor fusion, communication between clusters of

software intensive devices to coordinate activities,

communication to reach decisions between human-human, and

human-machine groups and others.

Examples of application-oriented services provided by

middleware are:

• validation of information acquired from different

sources, assigning tags to data items if necessary

• transformation (and compression) of validated (and

fused, or otherwise processed) information into the

interim format defined by the middleware, in our case

linking the information with an interactive digital map

• tracking the position of agents that are linked to the

middleware, and storing their position in the interim

format,

• keeping track of the access rights of all the agents linked

to the middleware and checking the rights during any

transaction

• remembering the preferred formats of messages, specific

subscriptions for information from the agents, and

processing capabilities of each involved agent

• delivering the subscribed information and satisfying all

the constraints and requirements imposed by the agents

Another set of services in the middleware is for intrinsic use

(for handling the interim data format), these services are

required to create, maintain, update, and partition according to

the subscriptions from, and position of the agents. In this

paper we use interactive digital map as an interim data

structure (see map server in Figure 2). This set of services is

organized as a multi-agent system. Dedicated agents compose

and decompose the designated area in a digital map into

respective parts, extract specific layers from parts of the map,

add and delete application oriented icons in the map as

required, search and link background data about the objects

and icons on the map, update the positions of mobile icons,

etc. See section IV of this paper for more details.

The peculiarity of this middleware is in its autonomous and

smart operation that pays attention to individual properties and

requirements of the clients, and in (situation sensitive) on-line

validation of the outcome of its services. This becomes

possible due to application of the “mediated interaction”

concept (see [6]). This concept is built on a situation-aware

interactive model of computation [7] and on a well-established

message exchange paradigm where consumer has to subscribe

to a message. During the subscription process the subscriber

specifies the properties to be guaranteed in the message (e.g.

time of computing the contents of the message, position of the

message sender, validity time of the message, etc).

IV. PILOT STUDIES AND TESTS

This section discusses case studies and experiments that

have been carried out on situational information fusion and

utilization by mobile agents. The devised systems can be

potentially applied in a system comprising only machines but

also in a mixed team of humans and machines. The case

studies presented in this section are:

Figure 2 Middleware for forming team situation awareness of agents

126

.

• Distributed detection, identification and tracking of

mobile objects

• Utilization of situational information by autonomous

mobile agents

The overviews of the case studies presented in the paper

describe the simulations while the physical implementations

involving embedded computing nodes directly interfaced to

the physical world and exchanging data between each other

are currently being developed.

A. Distributed detection & identification

1) Application scenario

In the first case study friendly vehicles operating in a

non-familiar environment opposing static and mobile

adversaries are considered. The friendly vehicles move on a

pre-specified path (a road) which may also have some

obstacles on it.

The exchange and utilization of situational information

by mobile entities is proposed with local and global

information being explicitly separated. Instead of situational

information being directly exchanged by the mobile platform

agents, each platform has a situation information mediator that

provides the situational information relevant to the given

mobile platform to the control layer of the mobile platform

when the mobile platform requires it. The architecture for

exchanging situational information is similar to the approach

presented in [6]. However instead of the approach suggested

in there the mediators do not interact with each other directly

but instead there is a central data store to which all the

mediators provide information and form where the mediators

fetch the information required by the mobile platform they

represent. The central data store was used to simplify the

implementation of the simulation, replacing the central data

store with direct communication links between the mediators

does not change the operation of individual mediators. All the

situational data generated by a mobile platform agent is

written to the data out data store of the agent. The mediator

selects the situational data relevant to other mobile agents and

writes that to the central situational data store from where

other mediators can fetch the data. In the same manner the

mediator fetches only the data relevant to the mobile platform

agent that it represents and writes data to the data in data store

from where the agent is able to read the data and select the

optimum output action in the current situation. The relation

between the agent and the mediator is depicted on Figure 3.

The constraints of the information required by a mobile

platform agent depend on the type of the mobile vehicle, the

state of the vehicle and the current location of the vehicle so

the mediator fetches only the information relevant to the

mobile platform at the current time.

2) Simulation scenario

The approach outlined above was validated in a

simulation involving autonomous collaborating vehicles. In

the simulation scenario several vehicles have to follow a road

to reach their objective – a destination specified by the user.

The terrain type on the path changes and if the terrain type is

unfavourable (e.g. loose sand or mud) the vehicle is slowed

down by the environment when the vehicle enters the

unfavourable terrain at a medium speed. If the vehicle enters

the unfavourable terrain at high speed it is not slowed down

but there are penalties introduced by the high speed in terms of

vehicle detectability. To reduce the probability of being

detected the normal speed of movement for the vehicles is

medium. In the scenario the information on the unfavourable

terrain is not known to vehicles before the simulation is started

so a vehicle that enters the unfavourable terrain at the normal

speed of movement (medium) is slowed down. However when

a vehicle detects the unfavourable terrain, it can share that

situational information with other vehicles. If the vehicle

obtains the information on the terrain type from another

vehicle it is able to speed up before entering the unfavourable

terrain and thereby avoid the slowdown.

The simulation is further complicated by adversaries in

the area, which may be mobile or non-mobile. If a vehicle is in

the proximity of an adversary and the speed of the mobile

agent is above a certain threshold (above medium) the

adversary “notices” the mobile agent and “kills it”. The

situational information regarding the adversaries can be shared

as well between the vehicles, giving the vehicles a chance to

slow down before approaching the adversary.

3) Simulation components

Every vehicle in the simulation is implemented as an

agent interfaced to a mediator as depicted on Figure 4. The

agent is responsible for the decision-making and the agent

mediator is responsible for the information exchange between

the agent and the external world. The agent in turn is

interfaced to the lower-level control functionality of the

vehicle, setting the control parameters for the vehicle, such as

immediate destination and speed.

To reduce the complexity of the implementation the

Figure 4 Screenshot of simulation scenario

Figure 3 Agent and mediator structure

127

.

situational information generated by the agents is propagated

to a central situational data store by the agent mediators as

depicted on Figure 5. Each agent mediator can then select the

data that is of interest to the agent it represents based on the

type and validity information of the situational data.

The agent notifies the mediator what data it requires (what are

the constraints on the validity intervals of the data) and the

mediator is responsible for acquiring the data to the agent.

Once the required data has been acquired it is placed in to the

data in data store from where the agent can fetch the data for

processing. In the same manner the situational data generated

by the agent is placed into the data out data store from where

the agent mediator can fetch the data and provide it to

mediators of other agents via the central situational data store

if such data has been requested.

Clearly the use of a central situational data store is a

simplification but it does not affect the behaviour of the

individual mediators much. If no central data store would be

used a mediator would have to interact directly with the other

mediators and check if the other mediators are able to provide

the data of interest.

4) Simulation implementation

The simulation is a combined Mathworks Matlab

Simulink / Stateflow simulation with a visual user interface for

validating the simulation results. A common coordinate

system is used in the simulation which allows identifying

objects by coordinates (among other parameters) and also

reconstructing the path of the vehicles from the coordinate log.

The following situation parameters are used in the

simulation i.e. the mobile agents detect, store and use the

following situation parameter values:

• unfavourable terrain start and end

• static enemy

• mobile enemy

The coordinates of the situational data and the timestamp

for the time when the value for situation parameter was

computed are stored for all the situational information items.

Each mediator selects the situational data based on the spatial

and temporal properties (where and when the parameter value

was valid) of the data. In this simulation also the validity

period of the data is evaluated by the consumer of the data, i.e.

the agent mediator discards the situational data record if the

timestamp of the record is too old.

The information stored for different types of situational

data varies depending on the data type. For example for the

unfavourable terrain type both the start and end coordinates of

the unfavourable terrain must be recorded in addition to the

timestamp. For an adversary the coordinates of the adversary,

the type of the adversary and the timestamp should be

recorded. Since the coordinates of the adversary can’t be

determined by a vehicle in the current scenario, the

coordinates of the vehicle that detected the adversary are

recorded at the time instant when the adversary was detected

(which is usually the time instance when the friendly vehicle

was “killed”). Of course the information must be interpreted

accordingly as the indirect information on the location of the

adversary adds uncertainty

5) Results

The simulations showed that the exchange of situational

information can improve the performance of the autonomous

vehicles. The concept of situational information mediator

showed to carry potential since it allows reducing the amount

of information presented to the vehicle agent. Vehicle agents

were able to make decisions based on situational information

collected by other vehicle agents and thereby improve the

probability of mission success – to reach the desired

destination.

A. Utilization of situational information by autonomous

mobile agents

The solution presented in this section was motivated by the

potential to apply it in the field of Wireless Sensor Networks.

Wireless sensor networks (WSN) are built up of autonomous

computing nodes with limited computational power and a

wireless communication interface. As the name suggests,

wireless sensor nodes are equipped with sensors which allow

them to sense physical phenomenon.

Typically, most WSN applications are built using a

centralized architecture where (preprocessed) data collected

by the WSN nodes is collected to one central point for

processing [9, 10]. The centralized approach offers in many

cases higher fidelity than the decentralized approach [10] but

it has also some drawbacks as is not very beneficial because of

high requirements on communication bandwidth as all data

required by the processing algorithms must be transported to

that central point. In addition the centralized approach has a

single point of failure which is not acceptable in some

applications.

An alternative would be to process the data in a distributed

manner, thereby spreading the computational load among the

computing nodes (which can be viewed as agents in such a

scenario) and decreasing the bandwidth requirements. The

decentralized approach has other drawbacks related to the

complexity of distributing the computation. The algorithm

used for data processing must be suitable for partitioning and

the computational algorithm must be able to handle the

inherent uncertainties present in a WSN network – the number

and location of computing nodes is not known before the

Figure 5 Exchange of situational information

128

.

computation is started so it is not possible to fix these

properties beforehand either at the time of design or at the

time of system deployment. This in turn means that the

relations between the fragments of the algorithm used cannot

be fixed but the interactions between algorithm fragments are

generated dynamically. The limited computational resources

available at WSN nodes pose another constraint as this limits

the complexities of the algorithms that can be applied.

1) Proposed detection scheme

The proposed detection and tracking scheme described in

this section separates the identification and tracking tasks. The

tracking algorithm uses the situation parameter values

computed by the identification algorithm (i.e. the object

classification) to improve the fidelity of the tracking task.

The proposed distributed tracking and identification system

consists of up to M sensing and computing agents where one

agent can assume the role of a super-agent handling the

processing of the results of the identification task generated by

other agents (Figure 6). Each agent is able to compute the

situation parameter values that indicate the presence of a pre-

specified object. The agents continuously monitor the

environment and perform the object identification task locally,

computing situation parameter values that reflect the

correlation of the signature of the detected object to known

object classes. The identification is performed by analyzing

the sound signal acquired via a microphone interfaced to the

agent. Fast Fourier transform is performed on acquired data

samples of predetermined length and sampling frequency.

Situation parameter values (the correlation coefficients) are

computed of this transform in respect to previously recorded

data samples of different objects of interest stored at the agent.

The situation parameter values vi = [c1i, c2i, …, cNi] (where

i is the number of the agent and N is the number of

distinguished classes) computed by individual agents are

passed to the selected super-agent where the situation

parameter values from several agents can be combined for

computing a higher level situation parameter value – an

identification of the observed object with the spatial validity

information associated with the situation parameters. Although

it may seem that the described approach employs a central

computing node this is true only to a certain extent – the

computations required for identification and localization of the

observed object need to performed by some node in the

system (making use of the data generated by the other nodes)

and distributing these computations is clearly inefficient.

However the sensor data is processed to a higher level of

abstraction at each individual node making the evaluation of

the situation distributed.

In order to ensure correctness of the object identification the

temporal consistency of the data items must be observed by

taking into account the temporal validity information of the

situation parameter values. At the super-agent, the object class

that has the highest correlation score is chosen typically and

the resulting decision (situation parameter value) is returned

back to the agents. The situation parameter value reflecting the

detected object type has a higher confidentiality value

associated with it as it combines observation results from

several agents. Based on the identification result (which can

be made locally or remotely) each agent then computes a

distance estimate based on signal energy/distance curve

specific to the given object class. In the case where the super-

agent is involved, the distance estimation information is

passed back to the super-agent. If there are at least three

independent distance estimates available at the super-agent,

with overlapping temporal validity intervals it is possible to

give an estimation of the location of the detected object using

the hyperbolic multilateration algorithm [11].

2) Practical experiment setup and acquired data

Practical experiments were carried out to validate the

approach outlined above. The approach was validated using a

PC and Mathworks MATLAB for data processing.

The equipment used in the experiments included three

Shure SM58 microphones connected to the Roland Edirol UA-

25EX audio interface attached to a PC. The maximum

bandwidth of the audio interface is 24 bits at 96 kHz so the

sampling rate and bit width (44.1 kHz and 16 bits) used for

recording sound samples in .wav format during the

experiments were well within the capabilities of the hardware.

The sound samples were recorded on a number of trial runs

with three mobile objects (forming three object classes) that

all moved at different speeds in the laboratory environment.

At this stage, the audio stream data is imported into

MATLAB and analyzed offline, using MATLAB built-in

functions (e.g. the implementation of fast Fourier transform)

as well as a number of user-written scripts.

The duration of the samples used for detection and

identification as well as for comparison was 0.1 seconds

(frequency resolution is thus 10Hz and the length of spectral

vectors is 2205). The same figure determines the minimum

interval at what coordinate determination of the object is

possible.

For object classification, fast Fourier transform is applied

to the measured signal. The resulting spectral vector p0 is

inserted into the matrix X = [p0, p1, …, pN], where pi, i = 1,

…, N are the spectral vectors of comparison samples.

The vector of correlation coefficients vi are obtained as

the first row (minus the first element) of the correlation matrix

R, whose (i, j)-th element is

central unit

agent 1 agent M

class

determination

correlation

coefficients distance

estimation

location

determination

c11, c21, ..., cN1

class

d1

c1M, c2M, ..., cNM

dM

xe

ye

Fig. 6 Distributed object identification and tracking system

129

.

jjii

ij

ijCC

C

⋅=ρ ,

where C is the (i, j)-th element of the covariation matrix of X.

It follows that the maximum element of Σvi points to the

most likely class for the unknown object (alternatively, if its

value is below a certain threshold, no class can be assigned).

Distance estimations to detected objects were computed

based on the amplitude of the received signal and the known

characteristics of the signal. The well known hyperbolic

multilateration positioning algorithm which performs quite

well if the distance estimations are correct [11] was used for

computing the position of the detected object.

3) Results of practical experiments

In the experiments, the objects were run on straight

trajectories. In the experiments, the estimated trajectory was

derived using a polynomial fitting function for a cluster of

estimated coordinates. Because the objects were run in straight

lines, the fitting polynomial is of the first order. The

classification accuracy for the tracked objects when the

situation parameter values computed by several agents were

combined was quite high, with some misclassified samples

only for class 3.

The work presented in this section shows that the

approach of distributed classification and positioning based on

audio signal analysis is feasible and can be used for object

identification and tracking. The accuracy of positioning still

needs improvement and is a subject for future research.

V. CONCLUSION

Achieving and maintaining situation awareness in a

distributed system containing both machines and humans is a

non-trivial task. The situational information generated by

individual agents that form the system must be synchronized

and the system design must ensure that the situation

parameters used by several or all system participants have

coherent values across the system. The paper presented the

principle of supplementing each situation parameter with

(temporal and spatial) validity information at the point where

the information is generated and using situational information

mediator coupled to each system agent for exchanging

situational information is an approach for achieving the

abovementioned objectives. The mediators coupled to

individual agents form a middleware that caters for the

exchange of situational information in a distributed system.

This paper also presented two case studies that implemented

some of the concepts presented in the paper. The first case

study showed how exchange of situational information is

beneficial in a mobile vehicle scenario and the second case

study presented a distributed detection and identification

scheme.

However many practical and theoretical problems must be

solved to implement such systems successfully, some of these

problems being closely related to the application domains. For

example the propagation and calculation of constraint and

validity information of situation parameter values; also the

description, validation and verification of systems’ behaviour

using situational information remains an unsolved task.

VI. ACKNOWLEDGMENTS

The work presented in this paper was partially supported

by Innovative Manufacturing Engineering Systems

Competence Centre IMECC that is co-financed by Enterprise

Estonia and European Union Regional Development Fund

(project EU30006).

REFERENCES

[1] H. Artman, C. Garbis. “Situation Awareness as Distributed Cognition”, in Proceedings of ECCE (European Conference on Cognitive

Ergonomics), Limerick, 1998, pp.?

[2] P. Salmon, N.A. Stanton, D.P. Jenkins, G.H. Walker, M.M.S. Young, A. Aujla. “What is really going on? Review, Critique and Extension of

Situation Awareness Theory”, in Engineering Psychology and Cognitive

Ergonomics, D. Harris, Ed., HCII2007, LNAI 4562, 2007, pp.407-416. [3] http://dictionary.reference.com/browse/situation, last visited 15.04.2009

[4] M. Endsley, G. Daniel, "Situation awareness analysis and

measurement", London : Lawrence Erlbum Associates, 2000 [5] Endsley M. R. Design and evaluation for situation awareness

enhancement, Proceedings of the Human Factors Society 32nd Annual

Meeting Vol. 1. - Santa Monica, CA : Human Factors Society, 1988. - pp. 97 - 101.

[6] Motus, L.; Meriste, M.; Preden, J. (2009). Towards Middleware Based

Situation Awareness. In: Military Communications Conference - 2009. MILCOM 2009, 18-21 Oct. 2009 : 5th IEEE Workshop on Situation

Management (SIMA 2009), Boston, 19-21 October. Boston, USA: IEEE

Operations Center, 2009, 1 - 7. [7] L. Motus, M. Meriste, W. Dosch “Time-awareness and Proactivity in

Models of Interactive Computation” Electronic Notes in Theoretical

Computer Science, vol. 141, (2005), 69-95, www.elsevier.com/locate/entcs

[8] L. Motus “Modeling metric time”, in B. Selic, L. Lavagno, G. Martin

(Eds), UML for Real: Design of Embedded Real-time Systems, Kluwer Academic Publ., Norwell, 2003, 205-220

[9] R. Viswanathan and P.K. Varshney, “Distributed detection with multiple sensors: Part I—fundamentals,” Proc. IEEE, vol. 85, no. 1, pp. 54–63,

Jan. 1997.

[10] J.-F. Chamberland, V.V Veeravalli, “Wireless Sensors in Distributed Detection Applications”, Signal Processing Magazine, IEEE, vol 24, no

3, pp 16-25, May 2007

[11] H.B. Lee, “Accuracy Limitations of Hyperbolic Multilateration Systems”, IEEE Transactions on Aerospace and Electronic Systems, vol

AES-11, no 1, pp 16-29, January 1975

[12] Ramage, J.; Sanz-Aranguez, P.; Campbell, J.; Cimen, T.; Crovella, L.; Dinc, M.; Kramer, I.; Martin, S.; Motus, L.; Preden, J.; Ravat, C.;

Robinson, M. (2010). “Design Considerations and Technologies for Air

Defence Systems”. NATO RTO Publications, SCI-181, 1 - 260. [13] Motus, L.; Michael G. R., “Timing Analysis of Real-Time Software”

Oxford : Elsevier, 1994. - ISBN 0080420257.

[14] Jakobson, G.; Buford, J.; Lewis, L.; "Situation Management: Basic Concepts and Approaches", Information Fusion and Geographic

Information Systems In Information Fusion and Geographic Information

Systems (2007), pp. 18-33. [15] Salmon, P. M.; Stanton, N. A.; Walker, G. H.; Jenkins, D. P., (2009)

“Distributed Situation Awareness Theory, Measurement and Application

to Teamwork", ISBN: 978-0-7546-7058-2, Ashgate Publishing

130