cvpr2007 distvisionproc chapter3 2 - pervasive.uni-klu.ac.at file3 cvpr 2007 short course...
TRANSCRIPT
1
Distributed Vision Processing Distributed Vision Processing in Smart Camera Networksin Smart Camera Networks
CVPRCVPR--0707
Hamid Aghajan, Stanford University, USAFrançois Berry, Univ. Blaise Pascal, FranceHorst Bischof, TU Graz, AustriaRichard Kleihorst, NXP Research, NetherlandsBernhard Rinner, Klagenfurt University, AustriaWayne Wolf, Princeton University, USA
March 18, 2007Minneapolis, USA
Course Website – http://wsnl.stanford.edu/cvpr07/index.php
CVPR 2007 Short Course 2Distributed Vision Processing in Smart Camera Networks
OutlineOutline
I. IntroductionII. Smart Camera Architectures
1. Wireless Smart Camera2. Smart Camera for Active Vision
III. Distributed Vision Algorithms1. Fusion Mechanisms2. Vision Network Algorithms
IV. Requirements and Case StudiesV. Outlook
2
Distributed Vision Processing Distributed Vision Processing in Smart Camera Networksin Smart Camera Networks
CVPRCVPR--0707
CHAPTER III:
Distributed Vision Algorithms
Hamid Aghajan, Wayne Wolf
CVPR 2007 Short Course 4Distributed Vision Processing in Smart Camera Networks
Vision Network AlgorithmsVision Network Algorithms
Contact: Wayne Wolf
wolf <AT> Princeton.Edu
3
CVPR 2007 Short Course 5Distributed Vision Processing in Smart Camera Networks
TopicsTopics
• Target modeling for distributed systems• Distributed state management and protocols• Calibration:
– Spatial– Temporal
CVPR 2007 Short Course 6Distributed Vision Processing in Smart Camera Networks
Distributed Vision AlgorithmsDistributed Vision Algorithms
• Starting points:– Single-camera algorithms– Multi-camera server-based algorithms
• New challenges:– No central repository– Communication delays
4
CVPR 2007 Short Course 7Distributed Vision Processing in Smart Camera Networks
Example: Distributed Gesture RecognitionExample: Distributed Gesture Recognition
• Lin et al: recognize gestures using graph model– Started with single-camera algorithm– Split block diagram and inserted protocol to
transform into distributed algorithm
CVPR 2007 Short Course 8Distributed Vision Processing in Smart Camera Networks
Token PassingToken Passing
5
CVPR 2007 Short Course 9Distributed Vision Processing in Smart Camera Networks
Algorithm PartitioningAlgorithm Partitioning
fixed migrating
CVPR 2007 Short Course 10Distributed Vision Processing in Smart Camera Networks
TokenToken--Passing ProtocolPassing Protocol
• Token represents ownership of high-level analysis
• Nodes periodically determine when token should move– Use target centroid as
heuristic• Handshake transfers
token
Low-levelanalysis
I owntarget?
I keeptarget?
noSend model
yesHigh-levelanalysis
yes
noHand off
token
Ack?no
6
CVPR 2007 Short Course 11Distributed Vision Processing in Smart Camera Networks
PeerPeer--toto--Peer TrackingPeer Tracking
• Velipasalar et al: Nodes must agree on target identity– In the presence of
multiple views• Need distributed
algorithm for agreement
Camera 1 Camera 2
Camera 3
CVPR 2007 Short Course 12Distributed Vision Processing in Smart Camera Networks
Distributed Tracking Agreement ProtocolDistributed Tracking Agreement Protocol
• Each node runs a tracker for each target in its field of view
• Nodes form ring for synchronization communication
• Nodes share tracker information every N frames
Camera 1 Camera 2
Camera 3T
T
7
CVPR 2007 Short Course 13Distributed Vision Processing in Smart Camera Networks
Label ConsistencyLabel Consistency
• Cameras must agree on label for target in shared field of view
• Negotiate agreement and label at next communication round after target crosses field of view line
CVPR 2007 Short Course 14Distributed Vision Processing in Smart Camera Networks
Target Position SharingTarget Position Sharing
• When target is occluded, node can get position from other nodes
• Provides fault tolerance
8
CVPR 2007 Short Course 15Distributed Vision Processing in Smart Camera Networks
Communication DiagramCommunication Diagram
CVPR 2007 Short Course 16Distributed Vision Processing in Smart Camera Networks
Communication ProtocolCommunication Protocol
Perform Local TrackingStore Incoming Msgs
Check Msgs For Synch PrevSend Synch Next
Process MsgsSubmit Requests
Receive RepliesModify Tracker Data
Check Msgs For Synch PrevSend Synch Next
Handle RequestsWait for/Send Synch All
Begin SynchPoint
End SynchPoint
9
CVPR 2007 Short Course 17Distributed Vision Processing in Smart Camera Networks
Target ModelingTarget Modeling• We use a fast blob tracker:
– Bounding box– Bhattacharya coefficient:
– qz is density function of the feature z representing the color of the target model
– pz(y) is the feature distribution of the FG blob centered at y
• All the usual problems:– Multiple views and models– Target merging and splitting
• We don’t build target merge/split into the protocol---handled camera-by-camera
CVPR 2007 Short Course 18Distributed Vision Processing in Smart Camera Networks
The synch_rate is 1 frame
Video: Indoor Setup 1Video: Indoor Setup 1
10
CVPR 2007 Short Course 19Distributed Vision Processing in Smart Camera Networks
The synch_rate is 30 frames
Video: Indoor Setup 2Video: Indoor Setup 2
CVPR 2007 Short Course 20Distributed Vision Processing in Smart Camera Networks
The synch_rate is 60 frames
Video: Indoor Setup 3Video: Indoor Setup 3
11
CVPR 2007 Short Course 21Distributed Vision Processing in Smart Camera Networks
Spatial CalibrationSpatial Calibration
• Need to find camera position/orientation, possibly camera internal parameters
• All the usual multi-camera problems• Algorithm must work without centralized
server
CVPR 2007 Short Course 22Distributed Vision Processing in Smart Camera Networks
Calibration using Belief PropagationCalibration using Belief Propagation
• Devarajan and Radke: Leverage dependencies introduced by camera geometries
• Belief propagation is used in sensor networks– Camera calibration introduces similarity
transforms that make the problem more difficult
12
CVPR 2007 Short Course 23Distributed Vision Processing in Smart Camera Networks
Vision GraphVision Graph• Nodes are connected if they
share enough visible points– Stable accurate estimate of
epipolar geometry can be obtained
• N(i): neighbors of node i• Constructing initial estimate:
– Estimate reconstruction from nucleus of i, N(i)
– Use RANSAC to reject outliers
– Adjust
CVPR 2007 Short Course 24Distributed Vision Processing in Smart Camera Networks
Basic Belief PropagationBasic Belief Propagation
• Each node updates its belief using messages from neighbors
• Update equations:
1
( ) /
( , ) ( ) ( )i
t tij i j i ki i i
k N i jY
m Y Y Y m Y dYφ −
∈
∝ Ψ ∏∫
( )
( ) ( ) ( )t ti i i ji i
j N i
b Y Y m Yφ∈
∝ ∏
Message from i to j at time t
Belief potential at i
Compatibility potential between i and j
13
CVPR 2007 Short Course 25Distributed Vision Processing in Smart Camera Networks
Camera CalibrationCamera Calibration
• Joint density of camera position based on camera observations can be factored:
• Two cameras have some variables Yi, Yj that must agree– Enforce using binary selector matrix C
1 2 1 2( , )
( , , , | , , , ) ( | ) ( , )M M i i i ji V i j E
p Y Y Y Z Z X p Z Y p Y Y∈ ∈
=∏ ∏K K
( , ) ( )i j ij i ji jP Y Y C Y C Yδ∝ −
CVPR 2007 Short Course 26Distributed Vision Processing in Smart Camera Networks
Update AlgorithmUpdate Algorithm
• Belief function:• Compatibility potential:
• Belief update computation:– Can be calculated iteratively in pairwise computations
( ) ( | )i i iY p Z Yφ ∝
( , ) ( )ij i j ij i ji jY Y C Y C Yψ δ∝ −
1 1 1 1 1
( ) ( )
[ ( ) ] ( ( ) )i j j ji i i i i i i
j N i j N i
μ μ μ− − − − −
∈ ∈
← Σ + Σ Σ + Σ∑ ∑
1 1 1
( )
[ ( ) ]i ji i i
j N i
− − −
∈
Σ ← Σ + Σ∑
14
CVPR 2007 Short Course 27Distributed Vision Processing in Smart Camera Networks
Temporal CalibrationTemporal Calibration
• Server-based systems are easy to synchronize:– Gen-lock– Cabling
• Peer-to-peer systems cannot be implicitly synchronized– Analog synchronization is
too expensive
server
CVPR 2007 Short Course 28Distributed Vision Processing in Smart Camera Networks
Timing Synchronization AlgorithmsTiming Synchronization Algorithms
• Lamport:– Read clock value of nodes periodically– Use average clock difference to adjust clock
• Lundelius:– Nodes broadcast timestamp at expected time– Nodes gather timestamp messages– Use medium value to adjust clock
• Halpern:– Nodes broadcast timestamp at expected time– Nodes update its clock when a faster timestamp is received
15
CVPR 2007 Short Course 29Distributed Vision Processing in Smart Camera Networks
System ConfigurationSystem Configuration
• Lin et al: Network configuration:
– Configuration 1: 6 cameras in a single network– Configuration 2 & 3: 10 cameras distributed in
several networks
CVPR 2007 Short Course 30Distributed Vision Processing in Smart Camera Networks
Lamport's Algorithm (cfg.2)
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.100
0.200
Synchronization Rounds
Diffe
renc
e w
ith re
al c
lock
(sec
)
Node 0 Node 1 Node 2 Node 3 Node 4
Node 5 Node 6 Node 7 Node 8 Node 9
Lamport's Algorithm (cfg.3)
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.100
0.200
Syncrhonization Rounds
Diffe
renc
e w
ith re
al c
lock
(sec
)
Node 0 Node 1 Node 2 Node 3 Node 4
Node 5 Node 6 Node 7 Node 8 Node 9
Simulation ResultsSimulation Results• Lamport’s algorithm:
– The 10 nodes can only be synchronized within 300 ms– Sub-networks can converge if initial difference is small– Clock value of routers might oscillate between average values of
neighboring sub-networks
16
CVPR 2007 Short Course 31Distributed Vision Processing in Smart Camera Networks
Simulation ResultsSimulation Results• Synchronization requirements
– Distributed gesture recognition system• Current: 15.23 frames/sec @ Pentium III 1.0 GHz• Goal: 30 frames/sec
– All three algorithms can achieve 15 frames/sec requirement in configuration 1
– Only Halpern’s algorithm fulfills the requirements in configuration 2 and 3– Lamport’s and Lundelius’ algorithms need O(n2) messages in each round– Halpern’s algorithm need amortized O(n) messages in each round– Halpern’s algorithm advantages:
• Simplest in computation• Most precise synchronization• Fewer message exchanges• Timestamps can be hidden within data messages
CVPR 2007 Short Course 32Distributed Vision Processing in Smart Camera Networks
ImageImage--based Temporal Calibrationbased Temporal Calibration
• Velipasalar: Correlate images sequences to find synchronizing offset
17
CVPR 2007 Short Course 33Distributed Vision Processing in Smart Camera Networks
Temporal Synchronization AlgorithmTemporal Synchronization Algorithm
• Each camera calculates position of its target(s) in other camera’s field-of-view
• Each camera finds a possible match in the frame sequence of other camera and identifies initial frame offset between cameras
• Perform a confidence check for each track pair• Refine the match
CVPR 2007 Short Course 34Distributed Vision Processing in Smart Camera Networks
Confidence Check and RefinementConfidence Check and Refinement
• Not all target identifications will be equally good– Confidence check weights match by confidence value
• After confidence check, frame offset is refined using local search
18
CVPR 2007 Short Course 35Distributed Vision Processing in Smart Camera Networks
Results without Final Refinement StepResults without Final Refinement Step
99.399.3799.899.7599.6799.599Accuracy (%)
99379550140130119999output
1000800500400300200100Ground truth
CVPR 2007 Short Course 36Distributed Vision Processing in Smart Camera Networks
ReferencesReferences
• Wayne Wolf, Burak Ozer, and Tiehan Lv, “Smart cameras as embedded systems,” IEEE Computer, 35(9) September 2002, pp. 48-53.
• Chang Hong Lin, Tiehan Lv, Wayne Wolf, and I. Burak Ozer, “A peer-to-peer architecture for distrib-uted real-time gesture recognition,” in Proceedings, International Conference on Multimedia and Exhibi-tion, IEEE, 2004, vol. 1, pp. 27-30.
• Senem Velipasalar, Wayne Wolf, "Frame-Level Temporal Calibration of Video Sequences from Unsynchronized Cameras by Using Projective Invariants", IEEE International Conference on Advanced Video and Signal Based Surveillance, Como, Italy, September 15-16, 2005.
• Senem Velipasalar, Jason. Schlessman, Cheng-Yao Chen, Wayne Wolf, and Jaswinder Pal Singh, “SCCS: a scalable clustered camera system for multiple object tracking communicating via Message Pass-ing Interface,” in Proceedings, International Conference on Multimedia and Expo,IEEE, 2006.
• Zhaolin Cheng, Dhanya Devarajan, and Richard J. Radke, Determining Vision Graphs for Distributed Camera Networks Using Feature Digests. EURASIP Journal on Advances in Signal Processing: Special Issue on Visual Sensor Networks, Volume 2007, Article ID 57034, 11 pages.
• Dhanya Devarajan and Richard J. Radke, Calibrating Distributed Camera Networks Using Belief Propagation. EURASIP Journal on Advances in Signal Processing: Special Issue on Visual Sensor Networks, Volume 2007, Article ID 60696, 10 pages.
• C. H. Lin and W. Wolf, “A case study in clock synchronization for distributed camera systems,”Proceedings of the SPIE, vol. 5683, January 2005.
19
CVPR 2007 Short Course 37Distributed Vision Processing in Smart Camera Networks
OutlineOutline
I. IntroductionII. Smart Camera Architectures
1. Wireless Smart Camera2. Smart Camera for Active Vision
III. Distributed Vision Algorithms1. Fusion Mechanisms2. Vision Network Algorithms
IV. Requirements and Case StudiesV. Outlook