istorama: a content-based image search engine and hierarchical triangulation of 3d surfaces

Digital Days 29/6/2001

ISTORAMA: A Content-Based Image Search Engine andHierarchical Triangulation of 3D Surfaces.

Dr. Ioannis Kompatsiaris

Centre for Research and Technology Hellas

Informatics and Telematics Institute

Thermi-Thessaloniki, Greece

ikom@iti.gr

Digital Days 29/6/2001

Outline• Introduction• Istorama architecture• K-Means with Connectivity Constraint Algorithm (KMCC)• Demo• Object/model based coding• Adaptive Triangulation and Progressive transmission• Reduced pyramid - quincunx sampling• Experimental results• Conclusions

Digital Days 29/6/2001

Need for efficient image search

• Huge number of images or databases of images

• Highly visual and graphical nature of the Web

• Text descriptors are not always efficient

• Greater flexibility with “content-based” access

• Queries which are more natural to humans

Digital Days 29/6/2001

Proposed approach

• Usually a description, a “signature” or a set indexes is created for the whole image

• Images usually contain different objects• Proposed approach: the image is first separated

into objects (segmentation)• Descriptors are created for each object• The user can search for a specific object

contained in images

Digital Days 29/6/2001

ISTORAMA architecture

Server

World Wide Web

Data BaseJDBC

Java Data Base Connection

User

PHP

Crawler - Spider

Indexing - Retrieval Algorithms

Digital Days 29/6/2001

The K-Means with Connectivity Constraint Algorithm (KMCC) I

• Based on K-Means algorithm• K-Means does not take into account spatial

information• In KMCC, the spatial proximity of each region is

also taken into account by defining a new spatial center and by integrating the K-Means with a component labeling procedure

• Automatic correction of the number of regions KK

Digital Days 29/6/2001

The K-Means with Connectivity Constraint Algorithm (KMCC) II

• Step1 K-Means is performed • Step2 Spatial centers are calculated

• Step3 Generalised distance

• Step 4 Component labeling LL connected regions

kCI

k

k

Sk

I AAkD

CSpCIpIp

22

1)(),(

kCS

Digital Days 29/6/2001

The K-Means with Connectivity Constraint Algorithm (KMCC) III

Digital Days 29/6/2001

Object descriptors

• Color, texture and spatial characteristics

• Color: histogram, 8 bins

• Spatial: (centroid),

• Shape: area, eccentricity

where λ1, λ2 are the two first eigenvalues

kCS

2

11

Digital Days 29/6/2001

Experimental Results (Synthetic)

Digital Days 29/6/2001

Experimental Results (Synthetic)

Digital Days 29/6/2001

Experimental Results

Digital Days 29/6/2001

Experimental Results (Claire)

Facial region

Moving object

Original sequenceFrames 1-

10

Segmentation

Digital Days 29/6/2001

Experimental results (Claire)

Digital Days 29/6/2001

Experimental results (table-tennis)

Original sequenceFrames 1-10

Digital Days 29/6/2001

Experimental results (table-tennis)

Segmentation

Moving objects

Digital Days 29/6/2001

Experimental results (Akiyo+Foreman)

Facial region

Facial region

Original sequenceFrames 1-

10

Original sequenceFrames 1-

10

Digital Days 29/6/2001

Conclusions

• K-means with spatial proximity algorithm• Multiple features segmentation• Higher order segmentation• Correspondence of objects between consequent

frames• Max-min criterion for automatic regularisation

parameters

Digital Days 29/6/2001

Future work

• Use of texture

• Indexing of video

• Integration with text descriptors

Digital Days 29/6/2001

• Triangular meshes of high quality are used in:

• Computer Aided Design • 3D representation of objects

(e.g. archaeological artifacts)• Animation and visual simulation• Entertainment (computer games)• Digital Terrain Modelling

Introduction

Digital Days 29/6/2001

Object/model-based coding

Digital Days 29/6/2001

Object/model-based coding

Digital Days 29/6/2001

Object/model-based coding

Digital Days 29/6/2001

Compression of finely detailed surfaces is necessary for:

• computation

• storage

• transmission

• display efficiency

Adaptive triangulation

Digital Days 29/6/2001

• Early, coarse approximations are refined though additional bits

Progressive transmission

Digital Days 29/6/2001

• Vertices removal and retriangulation [Schroeder] [Cohen]

• General mesh optimization process/function [Hoppe]

• Multiresolution analysis (MRA) [Lounsbery]

• Wavelets [Schroeder] [Gross]

• Progressive transmission [Schroeder] [Hoppe]

• Generalized triangle mesh representation [Deering]

Background

Digital Days 29/6/2001

Properties of the algorithm

• Efficient compression of the wireframe information• Simplification of the wireframe by adaptive

triangulation• Progressive transmission of the wireframe

information• Prioritised transmission of the wireframe• Straightforward correspondence between

successive scales

Digital Days 29/6/2001

Input surfaces

• Surface represented as a parametric function

in the parametric space

• determined by the position of a set of control points or nodes

• It allows for arbitrary, possibly closed wire-frame surfaces to be defined.

TvuzvuyvuxvuP ),(),,(),,(),(

2R

Tklklkl zyxlkr ,,),(

Digital Days 29/6/2001

Input surfaces

• The filters are applied to the 2D parametric representation of the surface as though it were a 2D image with intensity equal to

• Such surfaces include also:• depth images estimated from stereo pairs and• every surface that is homomorphic to a plane,

cylinder or torus

),( vuP

),( vuz

Digital Days 29/6/2001

Block diagram of the proposed procedure

Digital Days 29/6/2001

Reduced pyramid with quincunx sampling matrix

Digital Days 29/6/2001

Corresponding triangulation

Digital Days 29/6/2001

Optimal filters

• Optimal filters are determined by their Fourier transform:

• where is the power spectral density.

• Alternatively may be determined by the equation:

1,,1,0,1

021

Nr

eeeG M

iqj

rM

jrj

ri

T Mw

ww

Mk)(ig

wjr e

tktMMkpMtk

,)()(r

irir RgR

Digital Days 29/6/2001

Optimum bit allocation

• bits/vertex is assumed to be transmitted

• bits/vertex are allocated to each level using

• is the sum of error variances

BB rr 2

2

B

rB

2

r

ir

0

22

Digital Days 29/6/2001

Error prioritization

• The prediction errors corresponding to all predicted vertices are calculated and sorted with the vertices corresponding to higher errors being put first on the list

Higher Errors

Lower Errors

Digital Days 29/6/2001

Entropy estimation

• Entropy coding is used• The number of bits needed for error transmission

is the entropy of the errors • Using the quincunx sampling geometry at the

receiver, there is no need to transmit the exact co-ordinates of the position of each transmitted vertex

• The final cost of the transmission is the sum of the error entropy and the position entropy

Digital Days 29/6/2001

Adaptive Triangulation Procedure

• Synthesis stage of the QMVINT pyramid

• The vertex along with the vertices used to predict it are added to the mesh

• Handling of cracks

• Triangulation of the next vertex

)()(ˆ)( )()()( rk

rk

rk PerrPIPI

)(rkP

Digital Days 29/6/2001

Adaptive triangulation procedure

Digital Days 29/6/2001

Adaptive triangulation procedure

Digital Days 29/6/2001

Experimental results

• Original dense depth map and surface of the “Venus” data

Digital Days 29/6/2001

Experimental results

• 2569 vertices and 4006 triangles at level 2 MSE = 1.30

Digital Days 29/6/2001

Experimental results

• 7661 vertices and 11135 triangles at level 1 MSE = 1.30

Digital Days 29/6/2001

Experimental results

• 11416 vertices and 15827 triangles at level 0 MSE = 0.12

Digital Days 29/6/2001

Experimental results

Digital Days 29/6/2001

Conclusions

• Hierarchical representation of 3D surfaces using 3D adaptive triangular wireframes

• The variance of the error transmitted is minimised and therefore results to optimal compression of the wireframe information

• It produces a hierarchy where coarse meshes are as similar to their finer versions as is possible

Digital Days 29/6/2001

Conclusions

• The triangulation algorithm is integrated with a bit allocation procedure

• The number of nodes and triangles of the wireframe as well as the information needed for the transmission or storage of the wireframe are reduced simultaneously using a unified approach (QMVINT filtering)

• Precise correspondence between triangles at each level is achieved

Digital Days 29/6/2001

Future work

• Expansion and application directly to 3D surfaces

• Estimation of filters