MSU Fall 2019 1

Computing Motion from Images

Chapter 9 of S&S plus otherwork.

MSU Fall 2019 2

General topicsn Low level change detectionn Region tracking or matching over timen Interpretation of motionn MPEG compressionn Interpretation of scene changes in

videon Understanding human activites

MSU Fall 2019 3

Motion important to human vision

MSU Fall 2019 4

What’s moving: different cases

MSU Fall 2019 5

Image subtraction

Simple method to remove unchanging background from

moving regions.

MSU Fall 2019 6

Change detection for surveillance

MSU Fall 2019 7

Change detection by image subtraction

Closing = dilation+erosion

MSU Fall 2019 8

http://homepages.inf.ed.ac.uk/rbf/HIPR2/close.htm

MSU Fall 2019 9

What to do with regions of change?

n Discard small regionsn Discard regions of non interesting

featuresn Keep track of regions with interesting

featuresn Track in future frames from motion plus

component features

MSU Fall 2019 10

Some effects of camera motion that can cause problems

MSU Fall 2019 11

Motion field

MSU Fall 2019 12

FOE and FOC

Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.

Global Motion Compensation

13Seyed Morteza Safdarnejad, Yousef Atoum, Xiaoming Liu, “Temporally Robust Global Motion Compensation by Keypoint-based Congealing,” in ECCV 2016.

MSU Fall 2019 14

Gaming using a camera to recognize the player’s motion

Decathlete game

MSU Fall 2019 15

Decathlete gameCheap camera replaces usual mouse for input

Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.

MSU Fall 2019 16

Motion detection input device

Running (hands) Jumping (hands)

MSU Fall 2019 17

Motion analysis controls hurdling event (console)

•Top left shows video frame of player

•Middle left shows motion vectors from multiple frames•Center shows jumping patterns

MSU Fall 2019 18

Related workn Motion sensed by crude camerasn Person dances/gestures in spacen Kinect/Leap motion sensorsn System maps movement into musicn Creative environment?n Good exercise room?

MSU Fall 2019 19

Computing motion vectors from corresponding “points”

High energy neighborhoods are used to define points for matching

MSU Fall 2019 20

Match points between framesSuch large motions are unusual. Most systems track small motions.

MSU Fall 2019 21

Requirements for interest points

Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.

MSU Fall 2019 22

Interest = minimum directional variance

Used by Hans Moravec in his robot stereo vision system.

Interest points were used for stereo matching.

MSU Fall 2019 23

Detecting interest points in I1

MSU Fall 2019 24

Match points from I1 in I2

MSU Fall 2019 25

Search for best match of point P1 in nearby window of I2

For both motion and stereo, we have some constraints on where to search for a matching interest point.

MSU Fall 2019 26

Motion vectors clustered to show 3 coherent regions

All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin)

Motion coherence: points of same object tend to move in the same way

MSU Fall 2019 27

Two frames of aerial imagery

Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.

MSU Fall 2019 28

Can code frame N+d with displacments relative to frame N

n for each 16 x 16 block in the 2nd imagen find a closely matching block in the 1st

imagen replace the 16x16 intensities by the

location in the 1st image (dX, dY)n 256 bytes replaced by 2 bytes!n (If blocks differ too much, encode the

differences to be added.)

MSU Fall 2019 29

Frame approximation

Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)

MSU Fall 2019 30

Best matching blocks between video frames N+1 to N (motion vectors)

The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!

Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)

MSU Fall 2019 31

Motion coherence provides redundancy for compression

MPEG “motion compensation”represents motion of 16x16 pixels

blocks, NOT objects

MSU Fall 2019 32

MPEG represents blocks that move by the motion vector

MSU Fall 2019 33

MPEG has ‘I’, ‘P’, and ‘B’frames

MSU Fall 2019 34

Computing Image Flow

MSU Fall 2019 35

Motion Field & Optical Flow Fieldn Motion Field = Real world 3D motion n Optical Flow Field = Projection of the

motion field onto the 2d image3D motion vector

2D optical flow vector

( )vu,u =!

CCD

Slides from Lihi Zelnik-Manor

MSU Fall 2019 37

Assumptions

When does it break?

The screen is stationary yet displays motion

Homogeneous objects generate zero optical flow.

Fixed sphere. Changing light source.

Non-rigid texture motion

Slides from Lihi Zelnik-Manor

MSU Fall 2019 39

Image flow equation 1 of 2

MSU Fall 2019 40

Image flow equation 2 of 2

Estimating Optical Flown Assume the image intensity is constant

( )tyxI ,, ( )dttdyydxxI +++ ,,=

ITime = t Time = t+dt

( )dyydxx ++ ,( )yx,

Slides from Lihi Zelnik-Manor MSU Fall 2019 41

Brightness Constancy Equation( ) ( )dttdyydxxItyxI +++= ,,,,

( ) dttIdy

yIdx

xItyxI

¶¶+

¶¶+

¶¶+= ,,

First order Taylor Expansion

0=++ dtIdyIdxI tyx

Simplify notations:

Divide by dt and denote:

dtdxu =

dtdyv =

tyx IvIuI -=+Problem I: One equation, two unknowns

Slides from Lihi Zelnik-Manor MSU Fall 2019 42

Time t+dt

Problem II: “The Aperture Problem”

Time t

?Time t+dt

Where did the yellow point move to?We need additional constraints

n For points on a line of fixed intensity we can only recover the normal flow

Slides from Lihi Zelnik-Manor MSU Fall 2019 43

Use Local InformationSometimes enlarging the aperture can help

Slides from Lihi Zelnik-Manor MSU Fall 2019 44

Local smoothnessLucas Kanade (1984)

Assume constant (u,v) in small neighborhood

tyx IvIuI -=+ [ ] tyx Ivu

II -=úû

ùêë

é

úúú

û

ù

êêê

ë

é-=úû

ùêë

é

úúú

û

ù

êêê

ë

é

!!

2

1

22

11

t

t

yx

yx

II

vu

IIII

bA =u!Slides from Lihi Zelnik-Manor MSU Fall 2019 45

Lucas Kanade (1984)

bA =u!

( ) bAAA TT 1u

-=!

Goal: Minimize2u bA -!

bAAA TT =u!

2x2 2x1 2x1

Method: Least-Squares

Slides from Lihi Zelnik-Manor MSU Fall 2019 46

Lucas-Kanade Solution

úúû

ù

êêë

é=

åååå

2

2

yyx

yxxT

IIIIII

AA

We want this matrix to be invertible.i.e., no zero eigenvalues

( ) bAAA TT 1u

-=!

Slides from Lihi Zelnik-Manor MSU Fall 2019 47

Break-downsn Brightness constancy is not satisfied

n A point does not move like its neighbors n what is the ideal window size?

n The motion is not small (Taylor expansion doesn’t hold)

Correlation based methods

Regularization based methods

Use multi-scale estimation

Slides from Lihi Zelnik-Manor MSU Fall 2019 48

Multi-Scale Flow Estimation

image It-1 image I

Gaussian pyramid of image It Gaussian pyramid of image It+1

image It+1image Itu=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Slides from Lihi Zelnik-Manor MSU Fall 2019 49

SIFTflow

MSU Fall 2019 50

http://people.csail.mit.edu/celiu/SIFTflow/

Check out the Obstruction Free Photography work in 2016

MSU Fall 2019 51

Tracking several objects

Use assumptions of physics to compute multiple smooth paths.(work of Sethi and R. Jain PAMI)

MSU Fall 2019 52

MSU Fall 2019 53

Tracking in images over time

MSU Fall 2019 54

General constraints from physics

MSU Fall 2019 55

Other possible constraints

n Background statistics stablen Object color/texture/shape might

change slowly over framesn Might have knowledge of objects under

surveillancen Objects appear/disappear at boundary

of the frame

MSU Fall 2019 56

MSU Fall 2019 57

Sethi-Jain algorithm

MSU Fall 2019 58

MSU Fall 2019 59

Total smoothness of m paths

MSU Fall 2019 60

Greedy exchange algorithm

MSU Fall 2019 61

Example data structureTotal smoothness for trajectories of Figure 9.14

MSU Fall 2019 62

Example of domain specific tracking (Vera Bakic)Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback-control loop.

MSU Fall 2019 63

Segmentation of videos/movies

Segment into scenes, shots, specific actions, etc.

MSU Fall 2019 64

Types of changes in videos

MSU Fall 2019 65

Anchor person scene at left

Street scene for news story

Scene break

From Zhang et al 1993

How do we compute the scene change?

MSU Fall 2019 66

Histograms of frames across the scene change

Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.