Optical flow (motion Optical flow (motion vector) computationvector) computation

Course: Course: Computer Graphics and Image ProcessingComputer Graphics and Image Processing

Semester:Semester: Fall 2002Fall 2002

Presenter:Presenter: Nilesh Ghubade (nileshg@temple.edu)Nilesh Ghubade (nileshg@temple.edu)

Advisor: Advisor: Dr Longin Jan LateckiDr Longin Jan Latecki

Dept:Dept: Computer and Information Science, Computer and Information Science, Temple University, Philadelphia, PA-19122Temple University, Philadelphia, PA-19122

Motion AnalysisMotion AnalysisThree groups of motion-related problems:Three groups of motion-related problems:

1.1. Motion detectionMotion detection Registers any detected motion. Registers any detected motion. Single static camera.Single static camera. Used for security purposes.Used for security purposes.

2.2. Moving object detection and locationMoving object detection and location Determination of object trajectory.Determination of object trajectory. Static camera, moving objects OR Moving camera, Static camera, moving objects OR Moving camera,

static objects OR Both camera and objects moving.static objects OR Both camera and objects moving.3.3. Deriving 3D propertiesDeriving 3D properties

Use of set of 2D projections acquired at different Use of set of 2D projections acquired at different time instants of object motion.time instants of object motion.

Object motion Object motion assumptionsassumptions

Maximum velocity.Maximum velocity.

Small acceleration.Small acceleration.

Common motion of object points.Common motion of object points. Mutual correspondence.Mutual correspondence.

Cmax * dt

t0t1

t2

Differential motion Differential motion analysisanalysis

Simple subtraction of images acquired at Simple subtraction of images acquired at different instants in time makes motion different instants in time makes motion detection possible, assuming stationary detection possible, assuming stationary camera position and constant illumination.camera position and constant illumination.

Difference image is a Difference image is a binary image binary image subtract two consecutive images.subtract two consecutive images.

Cumulative difference image:Cumulative difference image: Reveals motion direction.Reveals motion direction. Time related motion properties.Time related motion properties. Slow motion and small object motion.Slow motion and small object motion.Constructed from sequence of ‘n’ images taking first Constructed from sequence of ‘n’ images taking first

image as the reference image.image as the reference image.

ExampleExampleMotion in front of a security camera.

Sobel filter edge detection.

Motion Detection- Sobel Motion Detection- Sobel filter filter 10 frames/second 15 frames/second

15 frames/second25 frames/second

Optical FlowOptical Flow Optical FlowOptical Flow reflects the image changes reflects the image changes

due to motion during a time interval due to motion during a time interval dt.dt. Optical flow fieldOptical flow field is the velocity field that is the velocity field that

represents the 3D motion of object points represents the 3D motion of object points across a 2D image.across a 2D image.

It should not be sensitive to illumination It should not be sensitive to illumination changes and motion of unimportant changes and motion of unimportant objects (e.g. shadows)objects (e.g. shadows)

Exceptions:Exceptions:1.1. Non-zero optical flowNon-zero optical flow fixed sphere illuminated by a fixed sphere illuminated by a

moving source.moving source.2.2. Zero optical flow Zero optical flow smooth sphere under constant smooth sphere under constant

illumination, although there is rotational motion and illumination, although there is rotational motion and true non-zero motion field.true non-zero motion field.

Optical Flow Optical Flow (continued…)(continued…)

Aim is to determine optical flow that Aim is to determine optical flow that corresponds with true motion field.corresponds with true motion field.

Necessary Necessary pre-conditionpre-condition of of subsequent higher level motion subsequent higher level motion processing processing stationary or moving stationary or moving camera.camera.

Provides tools to determine motion Provides tools to determine motion parameters, relative distances of parameters, relative distances of objects in the image etc..objects in the image etc..

Example:Example:t1 t2

AssumptionsAssumptionsOptical flow computation is based on Optical flow computation is based on

two assumptions:two assumptions:

1.1. The observed brightness of any The observed brightness of any object point is constant over time.object point is constant over time.

2.2. Nearby points in the image plane Nearby points in the image plane move in a similar manner (the move in a similar manner (the velocity smoothnessvelocity smoothness constraint). constraint).

Optical flow Optical flow computationcomputation

The optical flow field represented in the The optical flow field represented in the form of Velocity vector: form of Velocity vector:

Length of the vector determines the Length of the vector determines the magnitude of velocity.magnitude of velocity.

Direction of the vector determines the Direction of the vector determines the direction of motion.direction of motion.

Global optical flow estimation—Global optical flow estimation— Local constraints are propagated globally.Local constraints are propagated globally. But errors also propagate across the solution.But errors also propagate across the solution.

Local optical flow estimation—Local optical flow estimation— Divide image into smaller regions.Divide image into smaller regions. But inefficient in the areas where spatial But inefficient in the areas where spatial

gradients change slowly gradients change slowly here use global here use global method, neighboring image parts contribute.method, neighboring image parts contribute.

Forms of Forms of motionmotionTranslation at constant Translation at constant

distance from the distance from the observer.observer.

Set of parallel motion Set of parallel motion vectors.vectors.

Translation in depth Translation in depth relative to the relative to the observer.observer.

Set of vectors having Set of vectors having common focus of common focus of expansion.expansion.

Rotation at constant Rotation at constant distance from view distance from view axis.axis.

Set of concentric Set of concentric motion vectors.motion vectors.

Rotation of planar Rotation of planar object perpendicular to object perpendicular to the view axis.the view axis.

One or more sets of One or more sets of vectors starting from vectors starting from straight line straight line segments.segments.

RepresentationRepresentationLocate the position of a pixel (row,col) in the current image by computing shortest Euclidean distance with respect to 5-by-5 neighborhood in the next consecutive frame.

1616 1515 1414 1313 12121717 44 33 22 11111818 55 00 11 10101919 66 77 88 992020 2121 2222 2323 2424

ExperimentsExperiments

3-by-3 neighborhood

Experiments Experiments (contd…)(contd…)

5-by-5 neighborhood

Experiments Experiments (contd…)(contd…)

Experiments Experiments (contd…)(contd…)

Applications of optical Applications of optical flowflow

Object motion detection.Object motion detection.

Action recognition.Action recognition.

Active vision or structure of motion – Active vision or structure of motion – Reconstruction of 3D object by computing depth information.Reconstruction of 3D object by computing depth information. If you have distance (depth) maps, you can reconstruct If you have distance (depth) maps, you can reconstruct surface surface of of

the object.the object.

Facial expression recognition: referenceFacial expression recognition: referencehttp://athos.rutgers.edu/~decarlo/pubs/ijcv-face.pdfhttp://athos.rutgers.edu/~decarlo/pubs/ijcv-face.pdf

Thank youThank you