lecture 8

Image Indexing and Retrieval

Dr Crawford Revie(with thanks to Prof Fabio Crestani)

Lecture 8Multimedia Information Access

MIA Lecture 8 Crawford Revie (2006) 2

Outline of lecture

Why do we require image retrieval?Overview of techniquesExamples of systems

NB – if you print these slides out then you will probably want to use colour (or at least for some of them, 18-28 in particular don't make a lot of sense in b/w!)


Motivation and Application Areas

Management of Image ArchivesArt Galleries & MuseumsWWW Image Indexing

Science Databases (Medicine, Astronomy, Geography)Industry specific

Trademark DatabasesTextiles & FabricsAdvertisingArchitecture & Design


Approaches

There are 3 main approaches used in practice:1. Keyword based

manual / semi-automatic / automatic2. Based on visual properties

automatic3. Concept based

mostly manual (still in 'research' mode)


Keyword approach: indexing

Images are annotated using keywords

But:manual annotation is very expensive (as it is exceedingly time consuming)low level visual properties are almost impossible to index consistently using manual mark-upeven for 'high level' properties manual annotation is prone to subjectivity

Take a look at the Google Image Labeler 'game'http://images.google.com/imagelabeler


Keyword approach: retrieval

Since image description is textual, we use an almost straight-forward application of standard IR techniques (stop-word removal, stemming, indexing, etc.)

hypertext links have proved to be useful for retrieving images ("retrieval by browsing")thesauri and vocabularies can be more necessary here than in standard IR (e.g. AAT) – see also later discussion of structured keywords and concept-based retrieval


From manual to automatic keyword assignment

Keywords can be assigned to an image by analysing the text associated with the image

this includes the alt and caption attributes of the <img> and <table> HTML tagstext elsewhere on a web page containing the imagetext of the link pointing to the imageeven the name of the file containing the image


Automatic keywords

Why are thesepictures retrieved?


Automatic keywords

Check the text on the web page; the caption; and the filename


Structured keywords: using a database

Some systemsuse a DBMS to handle keywordsand searches


Visually based approaches

Often referred to as Content Based Image Retrieval (CBIR)

Similarity between query and documents is calculated based on visual features:

colourtextureshape

Visual features may be detected automatically or semi-automatically


Feature vectors

Images are represented using a set of feature vectors

),,( 1 inii IfIfIf K=

Queries are represented with the same set of feature vectors

),,( 1 inii QfQfQf K=


Vector features

Each feature has its own representation range of values, variability, etc.

Feature vectors may provide a “synthetic” view of a certain feature

In IR each word is represented by one feature exactly, but here one image characteristic may be represented by many features

similar to audio retrieval


Similarity functions

Need to choose similarity functions carefully:should be a good approximation to human perceptionof similarity between imagesshould have properties that help speed up computations

Different types of similarity evaluations may need to be combined to compare overall similarity (e.g. shape, colour, texture, etc.)


Colour based retrieval

Arguably easiest; earliest to be usedProcess is as follows:

represent image as a rectangular pixel raster (e.g. 1024 columns and 768 rows)represent each pixel as a quantified colour (e.g. 256 colours ranging from red through violet)count the number of pixels in each colour bin (this will produce a vector representation)compute vector similarity (e.g. using the normalised inner product)


Colour based matching

Let's compare some images retrieved using keyword: Godzilla


Colour histograms (for two samples)


Texture matching

Texture characterizes small-scale regularitycolour describes pixels, texture describes regions

Described by several types of featuressmoothness, periodicity, directionality, etc.

Match region size with image characteristicscomputed using filter banks, Gabor wavelets

Perform weighted vector space matchingusually in combination with a colour histogram


Texture matching


Image segmentation (shape)

Global techniques alone yield low precisioncolour & texture are better at characterising objects, not full images

Segment at colour and texture discontinuitieslike “flood fill” in Photoshop

Represent size shape & orientation of objectse.g. in Berkeley’s Blobworld we use ellipses

Represent relative position of objectse.g. angles between lines joining the centers

Segmentation allows us to perform object rotation and scale-invariant matching


"Flood fill" in Photoshop

More sophisticated techniques are needed


CBIR systems: Examples

Commercial systemsVirageQBIC

AcademicBlobworldVisualSeekChabotViper


QBIC

You can sketchan example of

what you are looking for


QBIC

Can you spotthe similarity to your 'query definition'?


Berkeley Blobworld

Can you spot the similarity to your 'query definition'?


Berkeley Blobworld


Blobworld Segmentation (1)


Blobworld Segmentation (2)


Viper: query by example

Provide examples

by indicatingrelevant images


Viper (QBE)

Retrieves similar

images


Concept based approach

Knowledge of the application domain is requirede.g. indexing of medical images requires knowledge of medicine(!) vocabulary + domain specific

System assigns concepts (index terms) to part of the image:

automatic concept assignment: very imprecise and ambiguous processmanual concept assignment: time-consuming and highly subjective process

Few experiments, semi-automatic seems best so far


Simple image retrieval is commercially availablecolour histograms, texture, limited shape information

Segmentation-based retrieval is still in the lab

Some way off:automatic identification and recognition of objects in images and videosconceptual image retrieval

The future?

lecture 8

Documents

filenamemia lecture

comimagelabelermia lecture

imagemia lecture

semiautomaticallymia

z manual annotation

z feature vectors

research modemia lecture

image description