Moving Object Detection and Moving Object Detection and Tracking for Intelligent Outdoor Tracking for Intelligent Outdoor

SurveillanceSurveillanceAssoc. Prof. Dr. Kanappan Palaniappan palaniappank@missouri.edu

Dr. Filiz Bunyak bunyak@missouri.eduDr. Sumit Nath naths@missouri.edu

Department of Computer ScienceUniversity of Missouri-Columbia

Visual Surveillance and MonitoringVisual Surveillance and Monitoring• Mounting video cameras is cheap, but finding available human resources to

observe the output is expensive.According to study of US Nat’l Institute of Justice:

• A person can not pay attention to more than 4 cameras.

• After only 20 minutes of watching and evaluating monitor screens, attention of most individuals falls below acceptable levels.

• Although surveillance cameras are already prevalent in banks, stores, and parking lots, video data currently is used only "after the fact".

What is needed

• Continuous 24-hour monitoring of surveillance video to alert security officers to a burglary in progress, or to a suspicious individual loitering in the parking lot, while there is still time to prevent the crime.

Intelligent SurveillanceIntelligent Surveillance A visual surveillance system combined with visual event detection methods to analyze movements, activities and high

level events occurring in an environment.

Event recognition module detects

unusual activities, behaviors, events

based on visual clues.

Sends an alarm to operators when a suspicious activity is

detected.

Visual Event Detection ApplicationsVisual Event Detection ApplicationsSurveillance and Monitoring:• Security (parking lots, airports, subway stations, banks, lobbies etc.)• Traffic (track vehicle movements and annotate action in traffic scenarios with natural

language verbs.)• Commercial (understanding customer behavior in stores)• Long-Term Analysis (statistics gathering for infrastructure change i.e. crowding

measurement)

Broadcast Video Indexing: Sports video indexing for newscasters and coaches.

Interactive Environments: environment that responds to the activity of occupants

Robotic Collaboration: robots that can effectively navigate their environment and interact with other people and robots.

Medical:• Event based analysis of cell motility• Gait analysis, etc.

Event TypesEvent TypesReal Time AlarmsLow level alarms:

Movement detectors, long term change detectors etc.

Feature based spatial alarms: Specific object detection in monitored areas

Behavior-related alarms: Anomalous trajectories, agitated behaviors, etc.

Complex event alarms: Detection of scenarios related to multiple relational events

Long Term and Large Scale Analysis Learning activity patterns of people or vehicles in a given environment over a long period of time can be used to:– retrieve events of interest – make projections– identify security holes– control the traffic or crowd– make infrastructure decisions– monitor behavior patterns in urban

environments

Issues in High Level Video AnalysisIssues in High Level Video Analysis

1-Analysis:• Segmentation of motion blobs (background models, shadow).• Object tracking (prediction, correspondence, occlusion

resolution etc.)

2-Representation: • Video object representations (shape, color descriptors,

geometric models). • High-level event representations.

3-Access: • Efficient data structures for high-dimensional feature space. • Efficient and expressive query interface for query manipulation.

Visual Event Detection FrameworkVisual Event Detection Framework

Feature Extraction

Motion Analysis

Event Detection

ObjectClassification

-Objects-Relationships

-Events

ContextObject, Scene &Event Libraries

Events

right turn

cross road

Constraints

Controlled Environment versus Controlled Environment versus Far-view Outdoor SurveillanceFar-view Outdoor Surveillance

Controlled Environment

(i.e. indoor)

Uncontrolled Environment

(i.e. far-view outdoor)

Illumination Controlled generally static except light switch which cause a global change.

Highly dynamic especially in cloudy days.

Shadows Smooth Darker and Sharper

Object Size Large Small (difficult to learn an appearance model)

Perspective Distortion

Low High

Color saturation High Low

Background Static Dynamic (wind etc.)

Type of motion Articulated Whole-body

Our Current CapabilitiesOur Current Capabilities

Moving Object Detection Moving Object Tracking

Sudden Illumination Change Detection

Trajectory Filtering and Discontinuity Resolution

Moving Cast Shadow Detection/Elimination

Our Current CapabilitiesOur Current Capabilities1. Moving Object Detection - Using Mixture of Gaussians

method or Flux tensors

2. Moving Cast Shadow Elimination

3. Sudden Illumination Change Detection

4. Moving Object Tracking – Multi-hypothesis testing using appearance and motion

5. Trajectory filtering - Temporal consistency check, spatio-temporal cluster check

6. Discontinuity resolution - Kalman filter, appearance model (color and spatial layout)

} Combined photometric invariants

Moving Object DetectionMoving Object DetectionGoal: Segment moving regions from the rest of the image

(background).

Rationale: Provide focus of attention for later processes such as tracking, classification, event detection/recognition.

Background SubtractionBackground Subtraction• By comparing incoming frames to a reference image (background

model), regions in the incoming frame that have significantly changed are located.

Feature Extraction

PostprocessingBG/FG

ClassificationComparisonPreprocessing

BG Modeling

BG/FG masks

Frames

BG model

Preprocessing-Spatial smoothing-Temporal smoothing-Color space conversions

Features-Luminance-Color-Edge maps-Albedo (reflectance)image-Intrinsic images-Region statistics

Comparison-Differentiation -Likelihood ratioing

Postprocessing-morphological filtering-connectivity analysis-color analysis-edge analysis-shadow elimination

Classification-Thresholding-Clustering

Problems with Basic Methods

Challenging Situations in Moving Challenging Situations in Moving Object DetectionObject Detection

1. Moved objects: A background object that moved should not be considered part of the foreground forever after.

2. Gradual illumination changes alter the appearance of the background (time of day).

3. Sudden illumination changes alter the appearance of the background (cloud movements).

4. Periodic movement of the background: Background may fluctuate, requiring models which can represent disjoint sets of pixel values (waving trees).

5. Camouflage: A foreground objects' pixel characteristics similar to modeled background.

6. Bootstrapping: A training period absent of foreground objects is not always available.

7. Foreground aperture: When an homogeneously colored object moves, change in interior pixels can not be detected.

8. Sleeping person: When a foreground object becomes motionless it cannot be distinguished from a background.

9. Waking person: When an object initially in the background moves, both the object and the background appear to change.

10. Shadows: Foreground objects' cast shadows appear different than modeled background.

Background ModelBackground Model

Color history of the specified pixel Color distribution of the specified pixel

inte

nsity

Mixture of Gaussians ModelThe recent history of each pixel, X(1),...,X(t), is modeled by a mixture of K Gaussian distributions. Each distribution is characterized by its

•mean μ,•variance σ2, •weight w (indicates what portion of the previous values did get assigned to this distribution).

Performance of Mixture of Performance of Mixture of Gaussians MethodGaussians Method

1. Moved objects √ 2. Gradual illumination changes √3. Sudden illumination changes X4. Periodic movement of the

background √ X5. Camouflage X6. Bootstrapping √7. Foreground aperture √8. Sleeping person √9. Waking person √10.Shadows X

Since MoG is adaptive & multi-modal, it is robust to:

• Gradual illumination changes• Repetitive motion of the background

(such as waving trees) • Slow moving objects• Introduction and removal of scene

objects (sleeping person & waking person problems)

– when something is allowed to become part of the background, the original background color remains in the mixture until it becomes the least probable and a new color is observed.

Moving Object Detection using Moving Object Detection using Flux TensorsFlux Tensors

Input sequence obtained from OTCBVS Benchmark Dataset Collection http://www.cse.ohio-state.edu/otcbvs-bench/

Color image sequence Thermal image sequence Moving Objects Detected using Flux Tensors

merges separateobjects

creates “new” objects

static shadow

Shadow ProblemShadow Problem

Shadow Detection by Combined Photometric Shadow Detection by Combined Photometric

Invariants for Improved Foreground SegmentationInvariants for Improved Foreground Segmentation

Moving Object Detection

Identification of Darker Regions

Normalized ColorComparison

Reflectance RatioComparison

CombinationPost

Processing

Shadow Detection

New Frame

FGmask

BG model

FGmask

Shadow Mask

FGmask

Shadow Mask

Combine the MasksCombine the MasksProblems with photometric invariants:• An invariant expression may not be unique to a particular

material.• There may be singularities and instabilities for particular

values. (normalized color is not reliable around black vertex).

For a robust result:• Combine results from two invariants based on two different

properties– Normalized color : spectral properties.– Reflectance ratio: spatial properties.– At shadow boundaries, same illuminant assumption fails.

different reflectance ratios for neighbor pixels misclassification of shadow pixels as foreground dilate shadow mask.

Example: Intelligent Room SequenceExample: Intelligent Room Sequence

Input Image Frame #100 MOG Model #1

MOG Model #2 MOG Model #3 MOG Model #4

Shadow MasksShadow Masks

Normalized Color MaskReflectance Ratio Mask

Shadow Mask Post processed shadow mask

Foreground & Shadow MasksForeground & Shadow Masks

Foreground Mask Post Processed Foreground Mask

Shadow Mask Post Processed Shadow Mask

Example: Walk-in SequenceExample: Walk-in Sequence

Input Frame Walk-in #14 Model 1

Model 2 Model 3 Model 4

Shadow MasksShadow Masks

Normalized Color Masks Reflectance Ratio Mask

Shadow Mask Shadow Mask Post Processed

Foreground & Shadow Masks

Foreground Mask Foreground Mask Post Processed

Shadow Mask Shadow Mask Post Processed

Sudden Illumination ChangesSudden Illumination Changes(Cloud Movements, Light switch etc.)(Cloud Movements, Light switch etc.)

Sudden illumination changes completely alter the color characteristics of the background, thus increase the deviation of background pixels from the background model in color or intensity based subtraction.

Result: • Drastic increase in false detection (in the worst case the whole

image appears as foreground).• This makes surveillance under partially cloudy days almost

impossible.

Moving Object TrackingMoving Object Tracking

Object States

Moving ObjectDetection &

Feature Extraction

Data Association(Correspondence)

Prediction

Update

Context

Tracking

Steps:

1. Predict locations of the current set of objects of interest.

2. Match predictions to actual measurements.

3. Update object states.

Tracking Tracking (as a Dynamic State Estimator)(as a Dynamic State Estimator)

Dynamic System State EstimatorMeasurement

System

System state

Measurements State estimate

Stateuncertainties

System Error Source•Agile motion•Distraction/clutter•Occlusion•Changes in lighting•Changes in pose•Shadow

(Object or background models are often inadequate or inaccurate))

Measurement Error Source•Camera noise•Grabber noise•Compression artifacts•Perspective projection

States•Position•Appearance

•Color •Shape•Texture etc.

•Support map

System noise Measurement noise

Our Tracking MethodOur Tracking Method• Detection-based• Probabilistic

Features Used in Data Association:– Proximity– Appearance

Data Association Strategy: Multi-hypothesis testingGating Strategies: Absolute and RelativeDiscontinuity Resolution:

– Prediction (Kalman filter) – Appearance models

Filtering: – Temporal consistency check– Spatio-temporal cluster check

Trajectory FilteringTrajectory Filtering

• Some artifacts can not be totally removed by image or object level processing.

• These artifacts produce spurious segments.

Temporal Consistency CheckTemporal Consistency CheckSource of the Problem: Segments resulting from • Temporarily fragmented parts of an object• Un-eliminated cast shadows

Effect: Short segments that split from or merge to a longer segment.

Proposed Solution: Pruning short split or merge segments by temporal consistency check.

Elimination of short disconnected segments are delayed until after discontinuity resolution.

Spatio-Temporal Cluster CheckSpatio-Temporal Cluster CheckSource of the Problem:• Repetitive motion of the background (i.e. moving branches or

their cast shadows).• Spectral reflections (i.e. reflections from car windshields).

Effect: Temporally consistent and spatially clustered trajectories.

Proposed Solution:• Average Displacement to Length Ratio (ADLR)• Diagonal to Length Ratio (DLR)

Discontinuity ResolutionDiscontinuity ResolutionDiscontinuities occur especially in low resolution outdoor sequences.

Source of the problem:• Temporarily undetected objects due to

– Low contrast– Partial or total occlusions

• Incorrect pruning in data association due to significant change in appearance or size caused by– Partial occlusion– Fragmentation

Discontinuity ResolutionDiscontinuity Resolution1. Define source and sink locations

where the objects are expected to appear and disappear.

2. Identify – Segdis :Segments disappearing

unexpectedly (at a non-sink location) -> possible start of a discontinuity.

– Segapp :Segments appearing unexpectedly (at a non-source location) -> possible end of a discontinuity.

3. Identify possible matches based on time constraint.

4. Use Kalman filter to predict future positions of disappearing and past positions of appearing segments.

5. Check direction and position consistencies on

– Disappearing segment– Appearing segment– Joining segment

6. Check Color similarity.7. Multiple possible matches for a single

disappearing segment-> select appearing segment starting earliest.

8. Multiple possible matches for a single appearing segment-> select disappearing segment ending latest.

9. Match-> appearing segment inherits disappearing segment’s label and propagates this new label to its children.

Challenges in TrackingChallenges in Trackingfor Visual Event Detectionfor Visual Event Detection

• Shadows -false detections, shape distortions, merges

• Sudden illumination changes (e.g. due to cloud movements) -difficulty in object detection especially in partly cloudy days

• Glare from specular surfaces (e.g. car windshields)-spurious detections and trajectory segments

• Perspective distortion (objects far away from the camera look smaller and appear to move slower)

-difficulty in filtering false detections• Occlusion

-discontinuities in trajectories• Poor video quality (low resolution, low color saturation)

-difficulty in moving object detection-difficulty in appearance modeling

Some Experimental Results-1Some Experimental Results-1

a) All segments b) Pruned segments

c) Predictions d) After discontinuity resolution

Some Experimental Results-2Some Experimental Results-2

a) All segments b) Pruned segments

c) Predictions d) After occlusion handlingUPS

Some Experimental Results-3Some Experimental Results-3

a) All segments b) Pruned segments

c) Predictions d) After discontinuity resolution

Potential Collaborations in Potential Collaborations in Visual Event DetectionVisual Event Detection

• New moving object detection methods– Flux tensor (especially in the

presence of global motion, clutter and illumination changes)

– Weather (i.e. snow, rain, wind) • Trajectory analysis

– Trajectory validation– Feature extraction – Trajectory annotation

• Extraction of primitive events based on

– Trajectory properties– Trajectory to trajectory interactions– Agent types

• Complex event detection/recognition through temporal combination of primitive events – Hierarchical approach

• Low-level : probabilistic methods

• High-level : structural methods

• Incorporation of learning to event modeling and recognition.

• Video event mining

Questions?Questions?