visual information retrieval chapter 1 introduction alberto del bimbo dipartimento di sistemi e...

Visual Information Retrieval

Chapter 1

Introduction

Alberto Del BimboDipartimento di Sistemi e InformaticaUniversita di Firenze

Firenze, Italy


• Information retrieval, image/video analysis and processing, pattern recognition and computer vision, visual data modeling and representation, multimedia database organization, multidimensional indexing, psychological modeling of user behavior, man-machine interaction and data visualization


• Types of associated information– content-independent metadata (CIM)

• format, author's name, date

– content-dependent metadata (CDepM)• low-level features concerned with perceptual facts:

color, texture, shape, spatial relationship, motion

– content-descriptive metadata (CDesM)• high-level content semantics: cloud, good weather, 白雲蒼狗


• First-generation visual information retrieval systems– CIM by alphanumeric strings, CDepM and

CDesM by keywords or scripts


– find images of paintings by Chagall with a blue background

• Select IMAGE# from PAINTINGS where PAINTER = "Chagall" and BACKGROUND = "blue"

– find images of paintings by Chagall with a girl in red dress and a blue background

• full text retrieval


– find images of paintings depicting similar figures in similar positions as in 收割景緻

• it is difficult for text to capture the perceptual saliency of some visual features

• text is not well suited for modeling perceptual similarity

• perception is mainly subjective, so is its text descriptions


• New-generation visual information retrieval systems– retrieval not only by concepts but also by

perception of visual contents• objective measurements of visual contents and

appropriate similarity models

• automatically extract features from raw data by image processing, pattern recognition, speech analysis and computer vision techniques


• Image retrieval– by perceptual features

• for each image in the database, a set of features (model parameters) are precomputed

• to query the image database– express the query through visual examples

» authored by the user

» extracted from image samples

– select features and ranges of features

– choose a similarity measure

• compute similarity degrees, ranking, relevance feedback


– system architecture• extraction of perceptual features (CDepM)

• extraction of high-level semantics (CDesM) from low-level features

• manual annotation of CIM and CDesM

• index structure

• graphical query tool

• retrieval engine

• visualization tool

• relevance feedback mechanism


• Video retrieval– special characteristics

• frames are linked together using editing effects

• color, texture, shape and position (camera or object) are changed in multiple frames

• richer semantics

• different types of video


– by structure• Figure 1.4

• frame: basic unit of information

• shot: elementary segment of video with perceptual continuity

• clip: set of frames with some semantic meaning

• scene: consecutive shots with simultaneous space, time and action

• episode: specific sequence of shot types such as a news episode

Visual Information Retrieval– by content

• perceptual properties, motion and type of an object• situations between objects• motion of camera• semantics of shots by color- or motion-induced sensations• semantics of scenes• stories• audio properties: dialogue, music or storytelling• textual information: caption or text recognized from video


– system architecture• extraction of shots and the associated semantics, key-

frames or mosaics

• extraction of scenes and stories

• manual annotation tool

• browsing/visualization tool– video summarization

• graphical query tool

• index structure

• retrieval engine


• 3D image and video retrieval

• WWW visual information searching– efficiency has to be emphasized due to limited

network bandwidth• operate in compressed domain

• visual summarization

• visualization at different levels of resolution


• Research directions– tools for automatic extraction of low-level

features– tools for automatic extraction of high-level

semantics– models for representing visual content– effective indexing– effective database models


– visual interfaces• allow querying and browsing

• allow querying by text and visual information

– similarity models• fit human similarity judgement

• psychological similarity models

– Web search– 3D image and video retrieval

visual information retrieval chapter 1 introduction alberto del bimbo dipartimento di sistemi e...

Documents