Two Examples Of Indoor And OutdoorTwo Examples Of Indoor And OutdoorSurveillance Systems:Surveillance Systems:

Motivation, Design, And TestingMotivation, Design, And Testing

Ioannis PavlidisIoannis Pavlidis

Vassilios MorellasVassilios Morellas

Honeywell LaboratoriesHoneywell Laboratories

Graduate Seminar in CISGraduate Seminar in CISVideo Processing and MiningVideo Processing and Mining

CIS 750 – Spring 2003CIS 750 – Spring 2003

Presented by:Presented by:

Ken GormanKen Gorman

AgendaAgenda

CCN – Cooperative Camera NetworkCCN – Cooperative Camera Network

DETER - Detection of Events for Threat DETER - Detection of Events for Threat Evaluation and RecognitionEvaluation and Recognition

Cooperative Camera Network (CCN)Cooperative Camera Network (CCN)

Network of cooperating cameras Network of cooperating cameras Controlled by computer vision softwareControlled by computer vision software Features:Features:

– mechanism for counting the people present in various mechanism for counting the people present in various parts of the buildingparts of the building

– An automatic or semi-automatic mechanism for tagging An automatic or semi-automatic mechanism for tagging people. people.

– Report tagged individuals whereabouts whenever they Report tagged individuals whereabouts whenever they are within the field of vieware within the field of view

Major ComponentsMajor Components

COTS Hardware & Software Set-UpCOTS Hardware & Software Set-Up Change DetectionChange Detection Counting People Counting People Tracking PeopleTracking People

State of the ArtState of the Art

Active badges Active badges – small, electronic devices worn by peoplesmall, electronic devices worn by people– transmit an ID signal to receivers placed around transmit an ID signal to receivers placed around

the buildingthe building– ID signal corresponds to the identity of the ID signal corresponds to the identity of the

badge’s wearerbadge’s wearer– received signals are used to compute the received signals are used to compute the

wearer’s locationwearer’s location

Badge ExamplesBadge Examples

Infrared-transmitting badges at Olivetti Infrared-transmitting badges at Olivetti Research and Xerox PARC, Research and Xerox PARC,

Olivetti ultrasonic badges at AT&T Olivetti ultrasonic badges at AT&T Laboratories in Cambridge, UK Laboratories in Cambridge, UK

Radio frequency tags from PinPointRadio frequency tags from PinPoint Wired and unwired motion trackers fromWired and unwired motion trackers from

– Ascension Technology Ascension Technology – PolhemusPolhemus

DisadvantagesDisadvantages

Consumers unwilling to wear badgesConsumers unwilling to wear badges CumbersomeCumbersome

AlternativesAlternatives

????????

CamerasCameras

Pro – Leaves users unencumberedPro – Leaves users unencumbered Cons – Not as reliable as badge methodsCons – Not as reliable as badge methods

Camera ArrangementCamera Arrangement

Overlapping Fields of ViewOverlapping Fields of View

FundamentalsFundamentals

Imaging Technologies for Surveillance Imaging Technologies for Surveillance SystemsSystems– Image SegmentationImage Segmentation– Tracking MechanismTracking Mechanism– Multi-Camera FusionMulti-Camera Fusion

Threat AssessmentThreat Assessment

Multi-Normal Pixel RepresentationMulti-Normal Pixel Representation

InitializationInitialization

Goal - provide statistically valid values for Goal - provide statistically valid values for the pixels corresponding to the scene.the pixels corresponding to the scene.

Starting point for the dynamic process of Starting point for the dynamic process of foreground and background awarenessforeground and background awareness

InitializationInitialization

Methods Used:Methods Used:– K-Means – better for plazas and mallsK-Means – better for plazas and malls– Expectation-Maximization – better for Expectation-Maximization – better for

changing weather conditions [1]changing weather conditions [1]

Image SegmentationImage Segmentation

Each pixel is considered as a mixture of Each pixel is considered as a mixture of five time-varying trivariate normal five time-varying trivariate normal distributionsdistributions

)(

15,,1,0

,~

5

1

5

13

weightssproportionmixingare

andi

where

Nx

iii

iiii

Image SegmentationImage Segmentation

The term represents a trivariate Normal The term represents a trivariate Normal distribution with vector mean and variance-distribution with vector mean and variance-covariance matrixcovariance matrix

iiN ,3

Image SegmentationImage Segmentation

The distributions are trivariate to account The distributions are trivariate to account for the three component colors (red, green, for the three component colors (red, green, and blue) of each pixel in the general case and blue) of each pixel in the general case of a color camera.of a color camera.

B

G

R

x

x

x

x

Image SegmentationImage Segmentation

For simplification, the variance-covariance For simplification, the variance-covariance matrix is assumed to be diagonal with matrix is assumed to be diagonal with xxRR,x,xGG,x,xBB , having identical variance within , having identical variance within each Normal component, but not across all each Normal component, but not across all componentscomponents

INx iBi

Gi

Ri

ii

25

13 ,~

Update CycleUpdate Cycle

Distributions are ordered based upon their Distributions are ordered based upon their weights.weights.

Member of a DistributionMember of a Distribution Distribution is in background or foregroundDistribution is in background or foreground Jeffreys [2] algorithm used for “matching” Jeffreys [2] algorithm used for “matching”

pixel to distributionpixel to distribution Distributions are UpdatedDistributions are Updated

MatchingMatching

We use the Jeffreys divergence measure (J) to determine whether the We use the Jeffreys divergence measure (J) to determine whether the incoming data point belongs to one of the existing distributionsincoming data point belongs to one of the existing distributions

The Jeffreys number measures how unlikely it is that one distribution The Jeffreys number measures how unlikely it is that one distribution (g) was drawn from the population represented by the other (f)(g) was drawn from the population represented by the other (f)

K* - prespecified cut-off valueK* - prespecified cut-off value

*,

,,

11

2

1

2

3,

min51

22

2

KgfJ

gfJgfJ

uuuugfJ

i

ii

i

gggg

gii

g

g

ii

Update – Match FoundUpdate – Match Found

Incoming pixel state is labeled either Incoming pixel state is labeled either background or foreground background or foreground

All the parameters of the matched All the parameters of the matched distribution are updated according to the distribution are updated according to the method of momentsmethod of moments

Only the weights of the other distributions Only the weights of the other distributions are updatedare updated

Update – No MatchUpdate – No Match

Incoming pixel state is labeled foreground Incoming pixel state is labeled foreground Last distribution in the ordered list is Last distribution in the ordered list is

replaced replaced All the parameters of the new distribution All the parameters of the new distribution

are updated are updated Only the weights of the other distributions Only the weights of the other distributions

are updatedare updated

Jeffrey’s AlgorithmJeffrey’s Algorithm

Jeffreys number measures how unlikely it is Jeffreys number measures how unlikely it is that one distribution (that one distribution (gg) was drawn from the ) was drawn from the population represented by the other (population represented by the other (ff).).

Broken CloudsBroken Clouds

Preferential, in orderPreferential, in order No ProferenceNo Proference

Segmentation of Moving ObjectsSegmentation of Moving Objects

The form of some of the distributions could The form of some of the distributions could changechange

Some of the foreground states could revert Some of the foreground states could revert to back-ground and vice versa.to back-ground and vice versa.

One of the existing distributions could be One of the existing distributions could be dropped and replaced with a new dropped and replaced with a new distribution.distribution.

Predictive TrackingPredictive Tracking

On-line segmentation of foreground pixelsOn-line segmentation of foreground pixels Calculation of blob centroidsCalculation of blob centroids Multiple-Hypotheses Tracking AlgorithmMultiple-Hypotheses Tracking Algorithm

Multiple-Hypotheses TrackingMultiple-Hypotheses Tracking

Recursive Bayesian probabilistic procedureRecursive Bayesian probabilistic procedure Does NOT commit early to trajectoryDoes NOT commit early to trajectory

Multiple Hypothesis TrackingMultiple Hypothesis Tracking

Multiple Hypothesis TrackingMultiple Hypothesis Tracking

Kalman filtering prediction based on constant Kalman filtering prediction based on constant velocity models velocity models

K-best hypothesis trajectory tree generation, K-best hypothesis trajectory tree generation, pruning and merging pruning and merging

Bayesian probability calculations for matching Bayesian probability calculations for matching input data to track hypothesisinput data to track hypothesis

See references [5] and [6] for exact algorithmSee references [5] and [6] for exact algorithm

Multi-Camera FusionMulti-Camera Fusion

Monitoring of large areas can only be Monitoring of large areas can only be accomplished using multiple camerasaccomplished using multiple cameras

Panoramic View is created by fusing Panoramic View is created by fusing individual camera FOVsindividual camera FOVs

Object motion registered against a global Object motion registered against a global coordinate systemcoordinate system

Multi-Camera FusionMulti-Camera Fusion

Optimal Coverage Scheme is createdOptimal Coverage Scheme is created Minimal use of cameras to minimize costMinimal use of cameras to minimize cost

Multi-Camera FusionMulti-Camera Fusion

Compute HOMOGRAPHY matrix H Compute HOMOGRAPHY matrix H between two cameras based on CoG of between two cameras based on CoG of moving objects appearing in the moving objects appearing in the overlapping areas of the two fields of view overlapping areas of the two fields of view

Requirement: Information exchange Requirement: Information exchange between respective computers (e.g., pixel between respective computers (e.g., pixel intensity data and CoG of moving objects in intensity data and CoG of moving objects in pixel coordinates) pixel coordinates)

Homography Matrices ComputationHomography Matrices Computation

Least Squares MethodLeast Squares Method Very popularVery popular Relatively simple Relatively simple Defined in Reference [6]Defined in Reference [6]

Homography MatrixHomography Matrix

Used Kanatani MethodUsed Kanatani Method Based on a statistical optimization theory for Based on a statistical optimization theory for

geometric computer visiongeometric computer vision Cures the deficiencies exhibited by Least-SquaresCures the deficiencies exhibited by Least-Squares

Kanatani MethodKanatani Method

Epipolar constraint may be violated by various Epipolar constraint may be violated by various noise sources due to the statistical nature of the noise sources due to the statistical nature of the imaging problemimaging problem

Multi-Camera FusionMulti-Camera Fusion

OO1 1 andand OO22 are Optical Centers are Optical Centers

P(x,y,z) is a point in the scene that falls in the P(x,y,z) is a point in the scene that falls in the common area between the two cameracommon area between the two camera

Vector OVector O11p, Op, O22q, and Oq, and O11OO2 2 are co-planarare co-planar

ReferencesReferences

Paolo Remagnino , et al (Editors). Paolo Remagnino , et al (Editors). Video-Based Surveillance Systems: Video-Based Surveillance Systems: Computer Vision and Distributed ProcessingComputer Vision and Distributed Processing. Kluwer Academic Publishers, . Kluwer Academic Publishers, 2002.2002.

http://www.htc.honeywell.com/projects/deter/http://www.htc.honeywell.com/projects/deter/ [1] - A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood [1] - A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood

from incomplete data via the em algorithm (with discussion),” J. Roy. Stat. from incomplete data via the em algorithm (with discussion),” J. Roy. Stat. Soc. B, vol. 39, pp. 1–38, 1977.Soc. B, vol. 39, pp. 1–38, 1977.

[2] - J. Lin, “Divergence measures based on the Shannon entropy,” IEEE [2] - J. Lin, “Divergence measures based on the Shannon entropy,” IEEE Trans. Inform. Theory, vol. 37, pp. 145–151, Jan. 1991.Trans. Inform. Theory, vol. 37, pp. 145–151, Jan. 1991.

[3] - C. Stauer and W.E.L. Grimson, “Adaptive background mixture models [3] - C. Stauer and W.E.L. Grimson, “Adaptive background mixture models for real-time tracking," in Proceedings 1999 IEEE Conference on Computer for real-time tracking," in Proceedings 1999 IEEE Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, June 23-25 1999, vol. 2, pp. Vision and Pattern Recognition, Fort Collins, CO, June 23-25 1999, vol. 2, pp. 246-252.246-252.

References (cont.)References (cont.)

[4] D. B. Reid, “An algorithm for tracking multiple targets”, IEEE [4] D. B. Reid, “An algorithm for tracking multiple targets”, IEEE Transactions on Automatic Control, vol. 24, pp. 843{854, 1979.Transactions on Automatic Control, vol. 24, pp. 843{854, 1979.

[5] I. J. Cox and S. L. Hingorani, “An efficient implementation of Reid's [5] I. J. Cox and S. L. Hingorani, “An efficient implementation of Reid's multiple hypothesis tracking algorithm and its evaluation for the purpose multiple hypothesis tracking algorithm and its evaluation for the purpose of visual tracking," IEEE Transactions on Pattern Analysis and Machine of visual tracking," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 2, pp. 138-150, 1996.Intelligence, vol. 18, no. 2, pp. 138-150, 1996.

[6] L. Lee, R. Romano, and G. Stein, “Monitoring activities from multiple [6] L. Lee, R. Romano, and G. Stein, “Monitoring activities from multiple video streams: Establishing a common coordinate frame," IEEE video streams: Establishing a common coordinate frame," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 758 { 767, 2000.8, pp. 758 { 767, 2000.