video object recognition chenyi chen. motion is important how important? let’s first look at...

Video Object Recognition

Chenyi Chen

Motion is important

• How important?• Let’s first look at “Visual Parsing After

Recovery From Blindness”• This is a real “vision” paper

Background

• Study how do three Indian patients (subjects) develop object recognition ability after long term blindness

• Give treatment to the subjects• During recovery, test on the subjects to see

how they perform on recognition tasks

Background

• The subjects are:• S.K.: age 29, male, born blindness, M.A. in

political science• J.A.: age 13, male, born blindness, never

received education• P.B.: age 7, male, born blindness• Control group: 4 normal sighted adults, similar

social background

Subjects’ parsing of static images

S.K. versus simple region partition algorithm

Dynamic information in object segregation

Motility rating and object recognition results

Follow-up testing after several months

What do we learn about developing visual parsing skill

• Early stages: integrative impairments, overfragmentation of images, compromise recognition performance

• However, motion effectively mitigates these integrative difficulties

• Motion appears to be instrumental both in segregating objects and in binding their constituents into representations for recognition

• So we have some insight of how people developing visual recognition ability

• Can we reproduce visual learning process on a robot?

• Let’s look at “Learning about Humans During the First 6 Minutes of Life”

A baby robot

Hypothesis in social development• The infant brain is particularly sensitive to the

presence of contingencies• The contingency drives the definition and

recognition of caregivers• Human faces become attractive because they

tend to occur in high contingency situations

Goal• Whether acoustic contingency information

(sound) would be sufficient for the robot to develop preferences for human faces

• If so, get a sense for the time scale of the learning problem

A baby robot

Settings

• The baby robot interacted with the lab members while recording image it saw

• Contingency detection engine analyzes sound signal for presence of contingencies

• Whether people were present is not specified• Whether people were of any particular

relevance is not specified • The only training label is the acoustic

contingency signal

Visual learning engine

• Probabilistic model• Only needs the images to be weakly labeled as

containing with high or low probability the object of interest, do not need to indicate where the objects are located on the image plane

• Implementable in a neural network• Run in real time at video frame rate

Hardware

• Plush baby doll• IEEE1394a webcam (capture images, only

grayscale images used for training)• Microphone (receive auditory signal)• Loudspeaker (baby makes excited noise)

Collecting data

• Record the auditory and visual signals for 88 minutes

• 2877 positive examples• 824 negative examples• Baby robot was placed in chair, stroller, and a

crib, with bright or dim lighting conditions• 9 persons interacted with the baby robot

Collecting data

• Select 34 positive examples and 200 negative examples for training (approx. 5 min 34 sec). The rest are used for testing

• The label is noisy

Results

• Evaluation: 2-Alternative Forced Choice Task (2AFC)

• 86.17% on the face detection task ( i.e., deciding which of two images contained a face)

• 89.7% correct on the contingency task (i.e., deciding which of two images was more likely to be associated with an auditory contingency)

• 92.3 % correct on the person detection task (i.e., deciding which image contained a person).

Results

• Examples images and their pixel-wise probability images

Results

• Infants showed a significant order of tracking preference in favor the face stimulus, followed by the scrambled stimulus, followed by the empty stimulus

• The robot reproduce the preference order

• Video usually contains more data for object detector training

• There is a domain difference between video and still image

• So “Analysing domain shift factors between videos and images for object detection” is necessary

Goal

• For a given target test domain (image or video), the performance of the detector depends on the domain it was trained on.

• Examine the reasons behind this performance gap.

• Train an object detector with samples either from still images or from video frames and then test the detector on both domains.

Dataset• Still images (VOC)• PASCAL VOC 2007• 10 class of moving objects chosen

Dataset

• Video frames (VID)• YouTube-Objects dataset• 10 classes of moving objects• Further annotated a few images to make the

dataset have comparable labels with VOC

Equalizing the number of samples per class

• Equalize the training samples of VOC and VID• 3097 in total over the 10 classes (Table. 1)• Only the equalized training sets are used• trainVOC• trainVID

Domain shift factors

• Spatial location accuracy: accuracy of bounding box

• Appearance diversity: consecutive frames in video are similar, thus less diverse

• Image quality: compression, motion blur etc. in video images

• Object detector: DPM

Spatial location accuracy

• Method of getting bounding box on video:• PRE: worst• FVS: better• Manual label: best

Spatial location accuracy• Reduce almost 4% of the gap (test on VOC)

Spatial location accuracy

• Equalization: using the ground truth (human labeled) bounding box on trainVID

Appearance diversity

• Near identical samples of an object in video


• Measure diversity:• Clustering (agglomerative clustering, L2

distance of HOG features): each cluster contains visually very similar samples

• Measure appearance diversity by counting the number of clusters

• Equalization: resample training sets so the number of images and clusters (of trainVOC and trainVID) are equal

Appearance diversity• Bridge the gap by 3.5% (test on VOC)

Image quality• Gradient energy: sum of gradient magnitudes in

HOG cells• Equalization: blur trainVOC by applying a

Gaussian filter

Image quality

Image quality

• Closes the gap by 1% (test on VOC)

Training-test set correlation• The final 7% performance gap• Domain-specific correlation/bias• Find nearest neighbor of testing images in

both training sets

Training-test set correlation

• According to nearest neighbor criterion• testVOC is most similar to trainVOC• testVID is most similar to trainVID• Such correlation leads to the final

performance gap

• Now we understand the gap between video domain and still image domain

• We still want to try transferring the knowledge learnt in video domain to image domain

• OK, then let’s look at “Learning Object Class Detectors from Weakly Annotated Video”

Benefits of video

• Easier to automatically segment the object from the background based on motion information

• Show significant appearances variations of an object

• Provide a large number of training images

Pipeline

• Each video contains an object as indicated by the video tag

• Automatically localize object in video clips, output one bounding box for each video

• Learn a detector from the video images and corresponding bounding boxes

• Domain adaptation• Test the detector on PASCAL 07 dataset

Localizing objects in real-world videos

• Extract shots of coherent motion from each video

• Robustly fit spatio-temporal bounding boxes (tube) to each shot (3~15 tubes per shot)

• Jointly select one tube per video by minimizing an energy function of similarity

• The selected tubes are the output of the algorithm (used to train a detector)


• Temporal partitioning into shot• Abrupt changes of the visual content of the

video• Thresholding color histogram differences in

consecutive frames


• Forming candidate tubes• Large-displacement optical flow (LDOF)• Clustering the dense point tracks based on the

similarity in their motion and proximity in location

• Fit spatio-temporal bounding box to each motion segment


• Example of tubes


• Joint selection of tubes• Energy function

• Each frame s has multiple candidate tubes

• Select one tube from the candidates for each frame s

• Selected tubes over all the frames• Coefficient α


• The pairwise potential

• Measure appearance dissimilarity• Encourage selecting tubes look similar over time• Tube ls, lq in two different frame s, q• Dissimilarity functions Δ (two types of features)

compare the appearance of the two tubes


• The unary potential

• Measure the cost of selecting tube ls in shot s• Δ: prefer tubes visually homogeneous• Γ: percentage of bounding-box perimeter

touching the border of image• Ω: objectness probability of the bounding box


• Minimization• Find the configuration L* of tubes over all

frames that minimizes energy E• L* is the final output and will be used to train

an object detector

Results

• Automatic tube selection • Compare with ground truth bounding box

(IoU>=50%)• The automatic tube selection technique

selects best available tube most of time

Learning a detector from the selected tubes

• Sampling bounding box• Reduce the number to manageable quantity• Select samples more likely to contain relevant

objects (using Γ and Ω exactly)• Train the object detector• DPM• SPM

Results

• Models:• VOC: model trained on PASCAL dataset• VMA: model trained on manually annotated

frames from video• VID: model trained on video with the

proposed automatic pipeline• Test on PASCAL dataset

Results

• Object detector without domain adaptation• The bounding boxes generated by the

proposed pipeline is closed to manually labeled ones

• Performance gap across domain is large

Domain adaptation

• Domain difference:• Higher HOG gradient energy in images• SVM based on GIST feature to distinguish

video from images, accuracy 83%

Domain adaptation

• Large quantity of video (source domain) training data, small number of PASCAL image (target domain) training data

• Adaptation methods:• All: directly train a single classifier using the

union of all available training data• Pred: use the output of the source classifier as

an additional feature for training the target classifier

Domain adaptation

• Adaptation methods (cont.):• Prior: the parameters of the source classifier

are used as a prior when learning the target classifier

• LinInt: first train two separate classifiers fs(x), ft(x) from the source and the target training data, and then linearly interpolate their predictions on new target data at test time

Results

• Object detector with domain adaptation• Improvement w.r.t. VOC model• Most method (combine VOC training data at early

stage) degrades performance• The LinInt is immune to negative transfer• Knowledge can be transferred from video to image

domain

• We can not only automatically output bounding boxes on video, but also automatically segment the video into background and foreground object

• Fast object segmentation in unconstrained video

Goal

• Propose an automatic technique for separating foreground objects from the background in a video

• Two main stages:• 1) Efficient initial foreground estimation• 2) Foreground-background labeling

refinement

Efficient initial foreground estimation

• Optical flow: supports large displacement and efficient GPU implementation

• Motion boundaries: • magnitude of the gradient of the optical flow

field• difference in direction between the motion of

pixel p and its neighbors N (if n is moving in a different direction than all its neighbors, it is likely to be a motion boundary)


• Problems with the motion boundaries:• Do not completely cover the whole object

boundary• Subject to false positive


• Inside-outside map (e.g. pixel level, 0: outside, 1: inside)

• Estimates whether a pixel is inside the object based on the point-in-polygon problem

• Any ray originating inside a closed curve intersects it an odd number of time. Any ray originating outside intersects it an even number of times.


• Inside-outside map (cont.)• Incomplete motion boundary• Shooting 8 rays spaced by 45 degrees• Majority vote for final decision• Optimized data structure for linear time

implementation when computing the map

Foreground-background labelling refinement

• Pixel labelling problem with two labels (foreground and background)

• Oversegment each frame into superpixels• Assign labels to superpixels• Superpixel i at frame t takes a label• All superpixels’ labes in all frames


• Energy function

• Output segmentation minimizes

• Minimize with graph-cut


• In the energy function:• A: appearance model, one for foreground, one for

background. Estimated based on the inside-outside map

• L: location model, propagate the per-frame inside-outside maps over time to build a more complete location prior

• V: spatial smoothness potential defined over edge• W: temporal smoothness potential defined over

edge

Experiment evaluation

• SegTrack dataset: 6 videos• Evaluation: number of wrongly labeled pixels

averaged over fall frames


• Considerably outperforms [6, 4, 18]• On par with [14], which is remarkable, given

that the proposed approach is simpler• [27] achieves lower, but is much slower• The SegTrack dataset is saturated


• YouTube-Objects dataset:• 10 diverse object classes• Ground truth bounding box provided for some

frames• Fit bounding box to largest connected

component of the segmentation output• Evaluation: PASCAL criterion (IoU>=0.5)


• Runtime:• Intel Core i7 2.0GHz machine• Given optical flow and superpixels, it takes 0.5

sec/frame• Considerably faster than the other strong

baselines (typically >100 sec/frame)

• With videos, we are able to extract objects, but we can even do something more crazy, which is revealing subtle movement of objects

• At last, let’s look at “Eulerian Video Magnification for Revealing Subtle Changes in the World”

Goal

• Reveal temporal variations in videos that are difficult or impossible to see with the naked eye and display them in an indicative manner

• Input standard video sequence• Output amplified signal to reveal hidden

information

Results

• https://www.youtube.com/watch?v=e9ASH8IBJ2U

https://www.youtube.com/watch?v=e9ASH8IBJ2U

https://www.youtube.com/watch?v=e9ASH8IBJ2U

It’s amazing, right?

How it works?

• First-order motion example:• : image intensity at position x and time t• : displacement function

• Motion magnification: synthesize the signal

• α: amplification factor

How it works?

• : applying a broadband temporal bandpass filter, picking out everything except f(x)

• Define• Then

• And

How it works?

How it works?

• General case: δ(t) is not entirely within the passband of the temporal filter B(x,t)

• δk(t): different temporal spectral components of δ(t), each will be attenuated by the filter by a factor ϒk

• Then, let

How it works?

• We need

• For high spatial frequencies and large amplification factor α, the first order Taylor expansion may not hold

• So, we have Higher spatial frequencies, smaller α

How it works?• Artifacts

How it works?

• Multiscale analysis• Scale-varying process• Assign different spatial frequencies with

different magnification factor α

Pipeline

Pipeline

• Spatial decomposition: video pyramid constructed by a separable binomial filter of size five

• Example temporal filter B(x,t), task specific

Thank you!

video object recognition chenyi chen. motion is important how important? let’s first look at...

Documents