motion tracking. image processing and computer vision: 82 introduction finding how objects have...

Motion Tracking

Image Processing and Computer Vision: 8 2

Introduction Finding how objects have moved in

an image sequence Movement in space Movement in image plane

Camera options Static camera, moving objects Moving camera, moving objects

Image Processing and Computer Vision: 8 3

Contents Acquiring targets

Image differencing Moving edge detector

Following targets Matching Minimum path curvature Model based methods Kalman filtering Condensation Hidden Markov Model

Image Processing and Computer Vision: 8 4

Camera calibration revisited Image to camera co-ordinate

transformation Intrinsic parameters

Camera to world co-ordinate transformation Extrinsic parameters

Image Processing and Computer Vision: 8 5

Camera to Image Co-ordinates Distortionless Camera

If no distortions uniform sampling

Co-ordinates linearly related offset and scale

s yo yyimy

sxoxximx

Image Processing and Computer Vision: 8 6

Camera to Image Co-ordinates Distorting Camera

Periphery is distorted

k2 = 0 is good enough

yxr

rkrkydy

rkrkxdx

222

42

211

42

211

Image Processing and Computer Vision: 8 7

Pinhole Camera

Image

ObjectOpticalcentre

Image and centre, object and centre are similar triangles.

f Z

Z

Yfy

Z

Xfx

Image Processing and Computer Vision: 8 8

World frame

Camera frame

translate androtate

Camera and World Co-ordinates

Image Processing and Computer Vision: 8 9

System architecture

Acquiretarget

Followtarget

Start End

Image Processing and Computer Vision: 8 10

Target acquisition Finding a target to follow

Differencing Moving edge detector

Image Processing and Computer Vision: 8 11

Change and Moving Object Detection Simplest method of detecting

change Compute differences between

Live and background images Adjacent images in a sequence

Image Processing and Computer Vision: 8 12

Image Differencing Differences due to

Moving object overlying static background

Moving object overlying another moving object

Moving object overlying same moving object

Random fluctuation of image data

Image Processing and Computer Vision: 8 13

Image Processing and Computer Vision: 8 14

Demo Inter frame differencing Difference from a background

pFinder

Image Processing and Computer Vision: 8 15

Background image Detecting true differences required

an accurate background Lighting changes? Camera movement?

Image Processing and Computer Vision: 8 16

Background image updates Periodically modify whole

background Will include changes in new

background Systematically incorporate non-

changed portions of image into background BBelse

LBBthenbackgroundLif

titi

tititi ti

1,,

,1,, ,

Image Processing and Computer Vision: 8 17

Critique Can identify changes in the image

data But what do the changes mean?

Need a second layer of processing To recognize changes Optical flow sidesteps this problem...

Image Processing and Computer Vision: 8 18

Target following Observing the positions of an

object or objects in a time sequence of images.

Object matching Minimum path curvature Model based methods

Image Processing and Computer Vision: 8 19

Matching Locate objects in each image Match objects between images Use methods of previous lectures

Image Processing and Computer Vision: 8 20

Minimum path curvature Observations of two objects in three

images

Image Processing and Computer Vision: 8 21

Image Processing and Computer Vision: 8 22

Image Processing and Computer Vision: 8 23

Image Processing and Computer Vision: 8 24

Image Processing and Computer Vision: 8 25

Which is “best” solution? One with overall straightest paths

For each solutionFor each path

Compute total curvature

Sum

Image Processing and Computer Vision: 8 26

Model based tracking Mathematical model of objects’

motions: position, velocity (speed, direction),

acceleration Can predict objects’ positions

Image Processing and Computer Vision: 8 27

System overview

MotionModel

VerifyLocation

PredictLocation

Update?

Image Processing and Computer Vision: 8 28

Newton’s laws

s = position u = velocity a = acceleration

all vector quantities measured in image co-ordinates

tautsts 2

0

21

Simple Motion Model

Image Processing and Computer Vision: 8 29

Prediction Can predict position at time t

knowing Position Velocity Acceleration

At t=0

Image Processing and Computer Vision: 8 30

Uncertainty If some error in a - a or u - u Then error in predicted position -

s

tautsts 20 2

1

Image Processing and Computer Vision: 8 31

Verification Is the object at the predicted

location? Matching

How to decide if object is found Search area

Where to look for object

Image Processing and Computer Vision: 8 32

Object Matching Compare

A small bitmap derived from the object vs.

Small regions of the image Matching?

Measure differences

Image Processing and Computer Vision: 8 33

Search Area: Why? and Where? Uncertainty in knowledge of model

parameters Limited accuracy of measurement Values might change between

measurements Define an area in which object

could be Centred on predicted location, s s

Image Processing and Computer Vision: 8 34

Update the Model? Is the object at the predicted

location? Yes

No change to model No

Model needs updating Kalman filter is a solution

Image Processing and Computer Vision: 8 35

Kalman filter

Mathematically rigorous methods of using uncertain measurements to update a model

Image Processing and Computer Vision: 8 36

Kalman filter notation Relates

Measurements y[k] e.g. positions

System state x[k] Position, velocity of object, etc

Observation matrix H[k] Relates system state to measurements

Evolution matrix A[k] Relates state of system between epochs

Measurement noise n[k] Process noise v[k]

Image Processing and Computer Vision: 8 37

Mathematically

k

ky

ky

kx

kx

ky

kx

kkkk

n

nxHy

0

0

1

0

0

0

0

1

How do observations relate to model?

Image Processing and Computer Vision: 8 38

Prediction of System State Relates system states at epochs k

and k+1

k

kky

kky

kkx

kkx

T

T

kky

kky

kkx

kkx

kkkkkk

v

vxAx

|

|

|

|

1000

100

0010

001

|1ˆ|1ˆ

|1ˆ|1ˆ

||1ˆ

Image Processing and Computer Vision: 8 39

Prediction of Observation From predicted system state,

estimate what observation should occur:

k

kky

kky

kkx

kkx

kky

kkx

kkkkkk

n

nxHy

|1ˆ|1ˆ

|1ˆ|1ˆ

0

0

1

0

0

0

0

1

|1ˆ

|1ˆ

|1ˆ|1ˆ

Image Processing and Computer Vision: 8 40

Updating the Model Predict/estimate a

measurement Make a measurement Predict state of model How does the new

measurement contribute to updating the model?

kk |1ˆ y

1ky kk |1ˆ x

Image Processing and Computer Vision: 8 41

Updating the Model

G is Kalman Gain Derived from A, H, v, n.

kkkkk

kkkkkkk

|1ˆ1|1

|11|1ˆ1|1ˆ

yyy

yGxx

covariancesystem

|||1

|1

|

C

GHCCC

nHHCHCG

kkkkkk

TkkTkk

Image Processing and Computer Vision: 8 42

Example Tracking two corners of a minimum

bounding box Matching using colour Image differencing to locate target

Image Processing and Computer Vision: 8 43

Condensation Tracking So far considered single motions What if movements change?

Bouncing ball Human movements

Use multiple models plus a model selection mechanism

Image Processing and Computer Vision: 8 44

Selection and Tracking

Occur simultaneously Maintain

A distribution of likely object positions plus weights

Predict Select N samples, predict locations

Verify Match predicted locations against image Update distributions

Image Processing and Computer Vision: 8 45

Tracking Using Hidden Markov Models Hidden Markov model describes

States occupied by a system Possible transitions between states Probabilities of transitions How to recognise a transition

Image Processing and Computer Vision: 8 46

Optic Flow Assume each pixel moves but does

not change intensity Pixel at location (x,y) in image 1 is

pixel at (x+x,y+y) in image 2. Optic flow associates displacement

vector with each pixel

Image Processing and Computer Vision: 8 47

t

Iu

y

Iv

x

I

t

I

t

y

y

I

t

x

x

I

tt

Iy

y

Ix

x

I

Ott

Iy

y

Ix

x

ItyxI

ttyyxxItyxI

x

0

0

0

,,

,,,,

2

Image Processing and Computer Vision: 8 48

Aperture Problem

I/ x horizontal difference I/ y vertical difference I/ t difference between imagesOne equation, two unknowns cannot solve equationCould solve for movement

perpendicular to gradient

Image Processing and Computer Vision: 8 49

Solution

Impose smoothness constraint:

Minimise total of sum of squares of velocity component gradients

image y

v

x

vy

u

x

u 2222

Image Processing and Computer Vision: 8 50

EED

EvEuEPD

PEvv

D

PEuu

yx

taverageyaveragex

yaverage

xaverage

222

is a constantIterate over a pair of frames, orover a sequence

Image Processing and Computer Vision: 8 51

Critique Assumptions

Linear variation of intensities Velocity changes smoothly

These are invalid Especially at object boundaries

Image Processing and Computer Vision: 8 52

Area Based Methods Match small regions in image 1

with small regions in image 2 Assume objects move but do not

deform Same formulation as for optical flow,

different areas involved….

tyxIttyyxxI ,,,,

Image Processing and Computer Vision: 8 53

Matching

Compute (x, y) by finding (u,v) that minimises

Then (u,v) = (x, y)

wi

wi

wj

wjtjyixIttjvyiuxIvuD ,,,,2

,

Image Processing and Computer Vision: 8 54

Summary Target acquisition

Image differencing Background model

Target following Matching Minimum path curvature Model based methods Optic flow

Image Processing and Computer Vision: 8 55

This “telephone” has too many shortcomings to be seriously considered as a means of communication. The device is of no value to us.Western Union internal memo, 1876