fuzzy systems lifelog management
DESCRIPTION
Case Study 2 A hierarchical Bayesian network for event recognition of human actions and interactions Sangho Park, J.K. Aggarwal Multimedia Systems, vol. 10, pp. 164-179, 2004. Fuzzy Systems Lifelog management. Outline. Overview Post estimation using a hierarchical Bayesian network - PowerPoint PPT PresentationTRANSCRIPT
Case Study 2
A hierarchical Bayesian network for event recognition
of human actions and interactionsSangho Park, J.K. Aggarwal
Multimedia Systems, vol. 10, pp. 164-179, 2004
Fuzzy SystemsLifelog management
• Overview
• Post estimation using a hierarchical Bayesian network
• Recognition by DBN
• Relative constraints
• Experiment
• Summary
Outline
2/35
Overview
• Recognition of human interaction– Applications: video surveillance, video-event annotation, virtual
reality, human-computer interaction, and robotics– Difficulty: Ambiguity caused by body articulation, loose clothing,
and mutual occlusion between body parts
• Previous work– A method to segment and track multiple body parts in two-
person interactions– Multilevel processing at pixel level, blob level, and object level
3/35
Motivation
• A methodology– To estimate body-part pose– To recognize different two-person
interactions including pointing, punching, standing hand-in-hand, pushing, and hugging
• System component– Bayesian network: estimate body
poses– Dynamic Bayesian network:
classify a sequence of body poses
4/35
Head Pose Estimation
• Environmental setup: lighting conditions, reflectance of light from the head
• Head pose: head’s 3D rotation angles, visible part
5/35
Head Pose Estimation: Example
• V1: angle of the vector• V2: ratio of the two ellipses• a, B: fixed
• P(V1=C|H2=B) = 0.18
6/35
Arm Pose Estimation
7/35
Arm Pose Estimation: Example
• P(V5=B|H3=C, H4) = 0.34
8/35
Leg Pose Estimation
9/35
Leg Pose Estimation: Example
10/35
Overall Body Pose Estimation (1)
11/35
Overall Body Pose Estimation (2)
12/35
Body-part Pose Recognition by DBN
• DBN hidden states– Q1: set of DBNs for legs {“both legs are together on the
ground”, “both legs are spread on the ground”, “and “one foot is moving in the air while the other is on the ground”}
– Q2: set of DBNs for the torso {“stationary”, “moving forward”, and “at least one arm gets withdrawn”}
– Q3: set of DBNs for arms {“both arms stay down”, “at least one arm stretches out”, and “at least one arm gets withdrawn”}
13/35
Interaction Recognition
• Whole-body pose: q1, q2, q3– {“stand still with arms down”, “move forward with arm(s)
stretched outward”, “move backward with arm(s) raised up”, “stand stationary while kicking with leg(s) raised up”, etc.}
• Two-person interaction– Subject = {torso, arm, leg} – Verb = {raise, lower, stretch, withdraw, stay, move forward,
move backward} – Object = {head, upper body, hand, lower body}
14/35
Relative Constraints: Spatial
• Examples– “standing hand-in-hand”: the torsos of the two persons be side
by side and facing in the same direction– “pointing at the opposite person”: the torsos of face one another
• Relative position and orientation– Gross level: proximity between two persons– Intermediate level: relative orientations of the torso poses
between the two persons– Detailed level: relative configuration of individual body parts
15/35
Relative Constraints: Temporal
• Interval temporal logic: before, meet, overlap, start, during, and finish
• Example– A pushing interaction
• Event A: a person moving forward with arms stretched outward toward the second person
• Event B: m of the second person
16/35
Experimental Results
• Human interaction (9): approaching, departing, pointing, standing hand-in-hand, shaking hands, hugging, punching, kicking, and pushing
• Image– 320*240 pixels– 15 fps– 6 pairs of different people with various clothing– 56(?) sequences (6*9)
17/35
Interaction Examples
18/35
BN’s Belief Changes
19/35
Performance of DBNs
• Leave-one-out-cross-validation– Training: 5 sequences, test: 1 sequence
• Accuracy: 78%– Approaching: 100– Departing: 100– Pointing: 67 similar interaction– Standing hand-in-hand: 83– Shaking hands: 100– Hugging: 50 occlusion– Punching: 67 similar interaction– Kicking: 83– Pushing: 50 similar interaction
20/35
Semantic Interpretation
21/35
Summary
• Contribution– A hierarchical framework for the recognition of two-person
interactions– BN for managing ambiguity in human interaction– A human-friendly vocabulary for high-level event description– Stochastic graphical model
• Future works– Extending the method to crowd behavior recognition– Incorporating various camera-view points– Recognizing more diverse interaction patterns
22/35
Case Study 3
Evolutionary Learning of Dynamic Probabilistic Models with Large Time LagsA. Tucker, X. Liu and A. Ogden-Swift
International Journal of Intelligent Systems,Vol. 16, no. 5, pp.621-646, 2001.
Fuzzy SystemsLifelog management
• Introduction
• Background
• Methodology
• Algorithm
• Evaluation
• Conclusions
Outline
24/35
Introduction
• Multivariate time-series (MTS)– A large number of interdependent variables– Large time lags between causes and effects (ex. Oil refinery
processing)
• Learning dynamic Bayesian networks– Not focused on learning models automatically – Focused on models with small time lags– Challenge task for large datasets with large possible time lags
25/35
Dynamic Bayesian Networks
• Bayesian networks– A set of n nodes {x1,…, xn}, representing the N variables in the
domain
– Each node, xi has a finite set of ri mutually exclusive states, vi1 to viri.
– Each node xi with a set of parents, πi has an associated probability table P(xi|πi).
• Dynamic Bayesian networks consist of BNs at differing time slices– Links over different time lags (non-contemporaneous links) and
within the same time lag (contemporaneous links)
Background
26/35
Learning Bayesian Network Structures
• K2/K3 algorithms– Use a greedy search which begins with an empty structure with no
links– Explores the effect of adding each of the possible links to the
current structure– K2 (a log likelihood metric) / K3 (a description length metric)
• Branch and Bound technique– Perform an efficient exhaustive search by stopping any further
exploration along a search path based on a bound
• Evolutionary methods– Larranaga et al. used a genetic algorithm with a repair operator to
remove cycles– Wong et al. used evolutionary programming with freeze, defrost
and a knowledge guided mutation (KGM)– Sahami used the mutual information
• Missing data management: Structural EM algorithm with Dempster ’s expectation maximization algorithm
Background
27/35
Evolutionary Learning Bayesian Network Structures
• Learning BNs involves scoring candidate network structures– Log likelihood– Description length metric of a network structure– Description length metric of encoding the dataset given that model
– n: number of nodes– ri: possible instantiations of the node– qi: possible instantiations of the parent nodes– Fij = ∑Fijk
– Fijk: frequency of occurrences in the dataset that the node xi takes on the value vik and the parent nodes πi take on the instantiation wij
Background
28/35
Representation
• Assume that a dynamic network contains no contemporaneous links
• n = N (# of variables at a single time slice) + Q (# of variables at previous time slice)
• A list of triples represents a possible networks (a,b,l)– a: the parent variable– b: the child variable– l: the time lag
Methodology
29/35
Useful Heuristics
• No contemporaneous links– Finding a good network structure finding a group of simple
tree structures
• LagMutation: Each mutation is based on a uniform distribution with mean equal to the present lag
• Autoregression links (a,a,1)
Methodology
30/35
Seeded GA for Search
• Seed the entire first population with links found from the single link analysis
– Using an approximate method to find a good list of single links rather than scoring the entire set
– Exploiting this knowledge in the first population by seeding it entirely with a random selection of good links
• EP method is particularly efficient at finding a good selection of links with good correlation
– An individual represents a single triples– Self-adapting parameters (SAP)
Methodology
31/35
Algorithm
32/35
Evaluation: Efficiency (1)
• Adapting static BN search algorithms for DBN search– K2/K3– The genetic algorithm– The evolutionary program
• Knowledge guided mutation (KGM)
33/35
Evaluation: Efficiency (2)
34/35
Evaluation: Structural Comparison
35/35
Conclusion
• Problem: Learning dynamic probabilistic models with large time lags
• Proposed method: EP-Seeded GA
• Future works: Discretisation & parameterisation
• Brainstorming – Mutually not-exclusive states Fuzzy BN– Hybrid of the GA and K2/K3
36/35