K O U R O S HM E S H G IPROGRESS REPORT TOPIC

Occlusion Aware Particle Filter Tracker to Handle Complex and

Persistent Occlusions usingMultiple Feature Fusion

To: Ishii Lab Members,Dr. Shin-ichi Maeda, Dr. Shigeuki Oba,

And Prof. Shin Ishii

9 MAY 2014

TRACKING APPLICATIONS

K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 2

MAIN APPLICATIONS

Surveillance Public Entertainment

Robotics Video Indexing

Action Recog.

TRACKING CHALLENGES

K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 3

MAIN CHALLENGES

Varying ScaleClutterDeformation

OcclusionIlluminationAbrupt Motion

Goal: Define p(Xt|Y1,…,Yt) given p(X1)

BAYESIAN TRACKING

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4

X1 X2 … Xt

Y1 Y2 … Yt

States: Target Location and Scale

Observations: Sensory Information

PARTICLE FILTER TR.INTRODUCTION• • • • • • • • • • • • • • • • • • • • • • • •

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 6

INPUT IMAGEFrame: t

RGB Domain

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 7

INPUT DEPTH MAPFrame: t

Depth Domain

Close Far

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 8

SENSORY INFORMATIONFrame: t

Sensory Information

, ,{ , }t rgb t d tI I I

Observation

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 9

STATE REPRESENTATION & OBSERVATION MODEL

Frame: t

State

{ , , , }t t t t tB x y w h{ }t tX B

( ; )t t tY g I B

w

h

(x,y)

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 0

FEATURES

Feature Set1{ ,..., }MF f f

Color

Shape Edge

Texture

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 1

TEMPLATEFrame: 1

Template1 1,1 ,1{ ,..., }M

f1 fj fM

1 ,1 1{ }Mi i

1 1{ ( )}if Y

… …

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 2

PARTICLES INITIALIZATIONFrame: 1

Particles, ,{ }k t k tX B1,2, ,k N

Initialized Overlapped

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 3

MOTION MODELFrame: t

Motion Model, , ,k t k t k tB B

→ t + 1

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 4

FEATURE EXTRACTIONFrame: t + 1

Feature Vectors , 1( )i k tf Y

f1 f2 fM

X1,t+1

X2,t+1

XN,t+1

…

…

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 1 5

FEATURE FUSIONFrame: t

Probability of Observation( | , )t t tp Y X ,1

( ( ) | , )Mi i t t i ti

p f Y B ( ( ) | , )t t tp f Y B ,1

( ),Mi i i t i ti

p D f Y ,1

exp ( ),Mi i i t i tiD f Y

,1

exp ( ),Mi i i t i tiD f Y

Each Feature(.)if(.)iD

i Indepen

dence

Assumptio

n

!

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 6

PROB. CALCULATIONFrame: t + 1

Particles

Brighter = More Probable

,k tp

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 7

TARGET ESTIMATIONFrame: t + 1

Feature Vectors 1 1

ˆ | ,...,t t tB B Y Y E

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 8

MODEL UPDATEFrame: t + 1

New Model

Model Update

1ˆ ˆ( ; )t t tY g I B

1ˆ ˆ( )t i tf Y

, 1 , 1

,

ˆ

(1 )i t i i t

i i t

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 9

RESAMPLINGFrame: t + 1

Proportional to Probability

1( | )t tp X X1

2

345

67

PARTICLE FILTER TR.CHALLENGES• • • • • • • • • • • • • • • • • • • • • • • •

PFT ISSUES

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 1

Appearance Changes

Model Drift

Deficient Feature Space

Uninformed Search

Optimized Feature Selection

Approximation of Target

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 2

APPEARANCE CHANGES

Same Color ObjectsBackground ClutterIllumination ChangeShadows, Shades

Use Depth!

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 3

MODEL DRIFT PROBLEM

Templates Corrupted! t

Handle Occlusion!(No Model Update During Them)

DEFICIENT FEATURE SPACE

* Local Optima of Feature Space* Feature Noise* Feature Failures

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 4

RegularizationNon-zero ValuesNormalization

PERSISTENT OCCLUSION

Particles Converge to Local Optima / Remains The Same Region

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 5

Advanced Motion Models(not always feasible)

Restart Tracking(slow occlusion recovery)

Expand Search Area!

DYNAMICS…

* The Search is not Directed* Neither of the Channels have Useful Information* Particles Should Scatter Away from Last Known Position

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 6

Occlusion!

OCCLUSIONdo not address occlusion explicitly

maintain a large set of hypotheses

computationally expensive

direct occlusion detection robust against partial & temp occ. persistent occ. hinder tracking

GENERATIVE MODELS DISCRIMINATIVE MODELS

Dynamic Occlusion: Pixels of other object close to camera

Scene Occlusion: Still objects are closer to camera than the target object

Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ

UPDATE MODEL FOR TARGET TYPE OF OCCLUSION IS IMPORTANT KEEP MEMORY VS. KEEP FOCUS ON THE TARGET

Combine

Them!

PTO partial occlusion SAO self- or articulation occlusion TFO temporal full occlusion - shorter than 3

frames PFO persistent full occlusion CPO complex partial occlusion - including “split

and merge” and permanent changes in a key attribute of a part of target

CFO complex full occlusion

OCCLUSION TYPES

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 8

[Zhao & Nevatia, 04] Occlusion Indicator: Ratio of FG/BKG

[Wu & Nevatia, 07] Handle Occlusion using Appearance Model

[de Villiers et al., 12] Switch Tracker in the case of Occlusion

[Song & Xiao, 13] Occlusion Indicator: New Peak in HOD or Reduction of the Size of Main Peak

LITERATURE REVIW

Many other papers handle occlusions as the by-product of their novel trackers

OCCLUSION AWARE PFTSOLUTION• • • • • • • • • • • • • • • • • • • • • • • •

Motion Model

Resampling

Target Estimation

Calculate Likelihood

PRO

POSE

D M

OD

IFIC

ATIO

N

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 1

Initialization

Model Update

Observation

Occlusion Flag?

Constant Likelihood

Occlusion Estimation

Occlusion Threshold>?

YES

YES

NO

NO

Occlusion Flag (for each particle)

Observation Model

No-Occlusion Particles Same as Before

Occlusion-Flagged Particles Uniform Distribution

OCCLUSION AWAREPARTICLE FILTER FRAMEWORK

( | ) ( | , , )t t t t t tp Y X p Y B Z ( | ) (1 ) ( | , 0, ) ( | , 1, )t t t t t t t t t t t tp Y X Z p Y B Z Z p Y B Z

,k tZ

( | , 1, ) 1t t t tp Y B Z

,1( | , 0, ) exp ( ),M

t t t t i i i t i tip Y B Z D f Y

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 2

Position Estimation of the Target

Occlusion State for the Next Box

TARGET ESTIMATION

1 1

, , , ,1

ˆ [ | ,..., ]

( | , , )

t t t occ

Nj t j t j t j t t occj

Z u Z Y Y

u Z p Y B Z

E

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 3

1

, , , ,

ˆ [ | ,..., , 0]

( | , 0, )t t t t

j t j t j t j t tj

B B Y Y Z

B p Y B Z

E

J '

1

0

1

0 a

( )u x

( )u x a0a x

x

Model Update (Separately for each Feature)

Modified Dynamics Model of Particle

UPDATE RULE

11

1 1

ˆ( ) ,( )

ˆ ˆ( ) (1 ) ( ) ,t t occ

t

t t t occ

f Zf

f Y f Z

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 4

1 1 1 1 1( | ) ( , | , ) ( | ) ( | )t t t t t t t t t tp X X p B Z B Z p B B p Z Z

OA-PF DYNAMICS

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 5

Occlusion!

OA-PF DYNAMICS

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 6

Occlusion!

GOTCHA!

OA-PF DYNAMICS

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 7

Quick Occlusion Recovery Low CPE

No Template Corruption

No Attraction to other Object/ Background

CO

LO

R

(HO

C)

TE

XT

UR

E

(LB

P)

ED

GE

(L

OG

)

2D PR

OJ.

(BE

TA)

3D SH

APE

(PC

L Σ)FEATURES

DE

PTH

(H

OD

)

GR

AD

IEN

T (H

OG

)K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 8

& DISCUSSIONRESULTS• • • • • • • • • • • • • • • • • • • • • • • •

Princeton Tracking Dataset

DATASET( )

5 Validation Video with Ground Truth95 Evaluation Video

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 0

EXPERIMENT

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 1

OAPFT (Proposed, with different feature sets)

OI+SVM (SVM tracker with Occlusion Indicator)

• State-of-the-art RGBD tracker

ACPF (Adaptive Color Particle Filter)

• Traditional Particle Filter tracker

STRUCK (Structured Output SVM Tracker)

• State-of-the-art RGB tracker, Successful for Occlusion Handling

PASCAL VOC: Overall Performance

CRITERIA I

1

1

*1

* *1 1 1

*1 1

*1 1

ˆ

ˆ ˆ, 0ˆ1 , 1ˆ1 ,

t

t

t

t t t

t t t

t t

B B

B B Z Z

S Z Z

Z Z

0 1ott oS t AUC

toSu

cces

sOverlap Threshold

0

1

1

Area Under Curve

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 2

RESULTS

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 4 3

1

1

Success Plot

Overlap Threshold

Succ

ess R

ate

1

1

Mean Central Point Error: Localization Success

Mean Scale Adaption Error

CRITERIA II

* 2 * 21

ˆˆ( ) ( )Tt t t tt

w w h hSAE

T

* 2 * 21

ˆ ˆ( ) ( )Tt t t tt

x x y yCPE

T

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 4

ˆˆ ˆ ˆ ˆ{ , , , }t t t t tB x y w h * * * * *{ , , , }t t t t t

B x y w h

Estimated Ground Truth

RESULTSCenter Positioning Error

400

50Frames

CPE

(pix

els)

RESULTSScale Adaptation Error

140

50Frames

SAE

(pix

els)

FP happens when a tracker doesn’t realize that the target is occluded.

MI happens when the target is visible but the tracker fails to track it as if the target is still in an occlusion state

MT the estimated bounding box has nothing in common with ground truth box

FPS execution time in frames per second

CRITERIA III

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 7

RESULTS

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 4 8

Tracker

AUC

CPE

SAE MI FP MT FP

SBCDEGST (proposed)

76.50

9.59

7.32 0.0 2.4 0.0 0.9

ACPF (Nummiaro ‘03)

27.55

90.38

35.27

12.6 0.0 31.

0 1.4

STRUCK (Hare ‘11)

46.67

68.74

26.61

12.6 0.0 64.

413.4

OI+SVM (Song ‘13)

69.15

9.68

12.04 0.4 20.

0 0.8 0.4

FUTURE WORKS

K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 9

More Resilient Features + Scale

Adaptation

Active Occlusion Handling

Measure the Confidence of

each Data Channel

Adaptive Model Update

QUESTIONS?Thank you for your time…