ruleml2012 - rule-based high-level situation recognition from incomplete tracking data

© Fraunhofer IOSB 1

Rule-Based High-Level Situation Recognition

from Incomplete Tracking Data

David Münch1, Joris IJsselmuiden1, Ann-Kristin Grosselfinger1,

Michael Arens1, and Rainer Stiefelhagen1,2

1 Fraunhofer IOSB, Germany, [email protected] 2 Karlsruhe Institute of Technology, Germany.

The 6th International Symposium on Rules - Rule ML2012,

Montpellier, France, August 27-29, 2012


Motivation

• Security systems: Persons

and their behavior are

the focus of attention:

Person centric analysis

• Threat detection

• Visual surveillance

• Activity logging

• Video search

• Driver Assistance Systems

Input data is incomplete

and noisy.

Motivation


Overview

• Cognitive Vision System as a whole.

• High-level knowledge and situation recognition.

• Handling Incomplete Data.

• Experiments.


Cognitive Vision System


Conceptual Layer – Conceptual Primitives Level

• Quantitative information (from Quantitative Layer) is transformed into primitive

conceptual knowledge (Logic predicates).

• Dictionary of basic rules.

• Mainly domain independent.

• Support of uncertainty and vagueness.

• The rules in the CPL are mostly concerned with spatial relations and temporal

relations on short time intervals.


Dictionary of basic rules

Dictionary of basic rules for every

domain.

Fuzzy Metric Temporal Logic

(FMTL):

Extension of first order logic by

notions of fuzziness, time, and

metrics on time.

Inference engine: F-LIMETTE


“Numbers” mapped to Concepts

Fuzzy membership functions 𝜇𝑠𝑝𝑒𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 for the subset {zero, small, normal,

high, very_high} of discrete conceptual speed values.

Figure from: H.-H. Nagel, Steps toward a Cognitive Vision System, 2004, AI Mag.


Conceptual Layer – Behavior Representation Level

How to represent the expected Situations?

• Knowledge represented in Situation Graph Trees.

• Graphically editable

• Easy to extend and edit

• Interpretable (vs. black box)

• Exhaustive situation recognition.


Behavior Representation Level

Basic logic predicates from Conceptual Primitives Level are aggregated and

structured in Situation Graph Trees (SGT)

high-level conceptual situations.

An SGT consists of situation schemes:

• Can be start and/or end nodes.

• Unique name.

• State scheme (Precondition).

• Action scheme (Postcondition).

Specialize a situation scheme:

• Prediction edges link to a possible subsequent situation scheme.

• Specialization edges link to more specific situation graphs in a hierarchical

structure.


Behavior Representation Level


Handling Incomplete Data – Low Level in Scene Domain Level

Time

Perfect input data.

Truth value


Handling Incomplete Data – Low Level in SDL

Time

Incomplete input data.

Truth value


Interpolation of input data:


Time

Truth value


Interpolation of input data:


Time


Handling Incomplete Data – High Level in BRL



What if is_open(trunk, Car) fails?


Hallucination (abduction) of missing evidence.


What if is_open(trunk, Car) fails?

Hallucinate is_open(trunk, Car) and continue!


VIRAT Video Dataset

Place: 0000:

Videos: 02 03 04 06

Place: 0002:

Videos: 00 06

VIRAT Video Dataset Release 1.0

Input data: annotated ground truth: persons and their situations.


Situations

1. Person loads object into car.

2. Person unloads object from car.

3. Person gets into car.

4. Person gets out of car.

VIRAT Video Dataset Release 1.0


• The annotated ground truth is regarded as complete information.

• Randomly make gaps of a distinct length into the data.

• Increase the amount of missing data in steps of 5%.

• Repeat each experiment several times.

Evaluation


Evaluation

Original,

unmodified

F-Score

Precision Recall

Gap size

5 seconds.

Gap size

5 seconds.

Gap size

5 seconds.


Evaluation

ROC-curves for video (d) with gap size 5 seconds.

With interpolation and hallucination. Without interpolation and hallucination.

TPR TPR

FPR FPR


Evaluation

Video (d),

F-Score Gap size

5 seconds.

Gap size

5 seconds.

Gap size

5 seconds.


• low level: interpolation of data and its uncertainty.

ordinary incomplete data.

• high level: extension of the situation recognition inference algorithm

(hallucination, abduction).

high-level incomplete data such as occlusions.

• Knowledge base for vehicle-centered situations.

• Runs in real-time on off-the-shelf hardware.

Conclusion


Finis.