Internet Video Search

Arnold W.M. Smeulders & Cees Snoek

CWI & UvA

Overview Image and Video Search

Lecture 1 visual search, the problem

color-spatial-textural-temporal features

measures and invariances

Lecture 2 descriptors

words and similarity

where and what

Lecture 3 data and metadata

performance

speed

1 Visual search, the problem

A brief history of television

From broadcasting to narrowcasting

…to thin casting

~1955 ~1985 ~2005

2008

2010

Any other purpose than tv?

Surveillance to alert events Forensics to find evidence / to protect misuse Social media to sort responses Safety to prevent terrorism Agriculture to sort fruit News to reuse archived footage Business to have efficient access eBusiness to mine consumer data Science to understand visual cognition Family “I have it somewhere on this disk”

How big? The answer from the web

The web is video

…as of May 2011

How big? The answer from

Yearly influx

15.000 hours of video

1 Pbyte per year

Next 6 years

137.200 hours of video

22.510 hours of film

2.900.000 photo’s

How big? Answer from the archive

Crowd-given search

What others say is in the video. We focus on what digital content says is in the video.

Problem 1: The variation

So many images of one thing: illumination background occlusion viewpoint, … This is the sensory gap.

Multimedia Archives

Suit Basketball

Table

Tree

US flag

Aircraft

Dog Tennis Mountain

Fire

Building

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

1101011011011011011011001101011011111001101011011111

Problem 2: What defines things?

Language

Machine

Problem 3: The many things

This is the model gap

Problem 4: The story of a video

This is the narrative gap

Problem 5: No shared intuition

This is the query-context gap

Query-by-keyword

Query-by-concept

Query-by-examples

What sources Query

Prediction

Find shots of people shaking hands

System 1: histogram matching

Histogram as a summary of color characteristics. This image cannot currently be displayed.

Swain and Ballard, IJCV 1991

1 Conclusion

As content grows, many applications of image search.

Deep cognitive and computer science problems.

With simple means one gets visually simple results.

2 Features

Light source

Source . reflection

Result

)()( λρλe

Object

)(λρ

)(λe

(R,G,B)

)()()(

)()()(

)()()(

=

∫

∫

∫

λ

λ

λ

λλλρλ

λλλρλ

λλλρλ

dfe

dfe

dfe

BGR

B

G

R

++

++

++

=

BGRB

BGRG

BGRR

bgr

(r, g, b) in (R,G,B)

Independent of shadow!

The sensation of spectra

Hue: dominant wavelength λ(EH)

Saturation: purity of the colour (EH - EW)/EH

Intensity: brightness of the colour EW

“white” “green”

EH

EW

Human perception combines (R,G,B) response of the eye in opponent colors

Maximizes perceived contrast!

The sensation of spectra: opponent

λ

λ

λ

−−

−

++

=

)2(41

)(21

PuperGreenBlueYellowLuminance

GRB

GR

BGR

Color Gaussian space

−−=

BGR

EEE

17.060.034.035.004.030.0

27.063.006.0

λλ

λ

Geusebroek PAMI 2002

Maximizes information content!

(E0,Eλ,Eλλ)-pdf

Color Gaussian space (R,G,B)-pdf

Matter body reflectance in (R,G,B)

Taxonomy of diff-image structure T-junction

These junctions later bring recognition

Corner

Junction

Highlight

The 2D Gabor function is:

)(222

2

22

21),( vyuxj

yx

eeyxh ++

−= πδ

πσTuning parameters: u, v, σ

Gabor texture

Manjunath and Ma on Gabor for texture in Fourier-space

Gabor texture

K-means cluster of RGB

K-means cluster Gabor opponent

Hoang ECCV 2002

Gabor GIST descriptor

Calculate Gabor responses locally

Create histograms as before

Distinguishes things like naturalness, openness,

roughness, expansion, and ruggedness

Olivia IJCV 2001 Slide credit: James Hays and Alexei Efros

Receptive field in f(x,t)

Gaussian equivalent over x and t:

zero order first order t

Burghouts TIP 2006

Gaussians measure differentials

Taylor expansion at x

For discretely sampled signal use the Gaussians The preferred brand of filters: separable by dimension rotation symmetric no new maxima fast implementations.

Receptive fields: overview

All observables up to first order color, second order spatial scales, eight frequency bands & first order in t.

System 2: Blobworld, textured world

Group blobs based on color and Tamura texture

User specifies query blob and features

System returns images with similar regions

Carson PAMI 2002

2 Conclusion

Powerful features capture uniqueness.

A large set is needed for open-ended search.

The Gauss family is the preferred brand of filters.

Fast recursive implementation:

Geusebroek, Van de Weijer & Smeulders 2002

3 Measures and invariances

There are a million appearances to one object The same part of the same shoe does not have the same appearance in the image. This is the sensory gap. Remove unwanted variance as early as you can.

The need for invariance

A feature g is invariant under condition (transform)

caused by accidental conditions at the time of recording,

iff g observed on equal objects and is constant:

Invariance: definition

Quiz: scale invariant detection

What properties are invariant to observation scale?

surface albedo scene & viewpoint invariant

illumination scene dependent

object surface normal object shape variant

illumination direction scene dependent

viewer’s direction viewpoint variant

sensor sensitivity scene dependent

v

)(λCf

sn

)(λe)(λbc

λλλλλλλλλλ∫∫ += dfcevsnmdfcesnmC CssCbb )()()(),,()()()(),(

Color invariance

Matter body reflectance in E

E space C space

C is viewpoint invariant

Gevers TIP 2000

,,(3 BGRc},max{

arctan),,(2 BRGBGRc =

},max{arctan),,(1 BG

RBGRc =

Hue is viewpoint invariant

H = arctan 3 𝐺−𝐵𝑅−𝐺 + 𝑅−𝐵

, H is a scalar

Differential invariants C’, W’, M’

C’ is for matte objects and uneven white light:

EEC λ

λ =

2EEEEEC

EEC

xxx

λλλ

λλλλ

−=

=

W’ is for matte planar objects and even white light:

EEW x

x =E

EW xx

λλ =

M’ is for matte objects and monochromatic light: Geusebroek PAMI 2002 2E

EEEEN xxx

λλλ

−=

shadows shading highlights ill. intensity ill. color E - - - - -

H + + + + - W & W’ - + - + - C & C’ + + - + - M & M’ + + - + + L + + + + - E 990 H 315 Retained from 1000 colors σ = 3: W’ 995 C’ 850 M’ 900

Retained discrimination

Geusebroek PAMI 2003

3 Conclusion

Know your variances and invariants.

Good invariant features make algorithms simple.