istorama: a content-based image search engine and hierarchical triangulation of 3d surfaces
DESCRIPTION
ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces. Dr. Ioannis Kompatsiaris Centre for Research and Technology Hellas Informatics and Telematics Institute Thermi-Thessaloniki, Greece [email protected]. Outline. Introduction Istorama architecture - PowerPoint PPT PresentationTRANSCRIPT
Digital Days 29/6/2001
ISTORAMA: A Content-Based Image Search Engine andHierarchical Triangulation of 3D Surfaces.
Dr. Ioannis Kompatsiaris
Centre for Research and Technology Hellas
Informatics and Telematics Institute
Thermi-Thessaloniki, Greece
Digital Days 29/6/2001
Outline• Introduction• Istorama architecture• K-Means with Connectivity Constraint Algorithm (KMCC)• Demo• Object/model based coding• Adaptive Triangulation and Progressive transmission• Reduced pyramid - quincunx sampling• Experimental results• Conclusions
Digital Days 29/6/2001
Need for efficient image search
• Huge number of images or databases of images
• Highly visual and graphical nature of the Web
• Text descriptors are not always efficient
• Greater flexibility with “content-based” access
• Queries which are more natural to humans
Digital Days 29/6/2001
Proposed approach
• Usually a description, a “signature” or a set indexes is created for the whole image
• Images usually contain different objects• Proposed approach: the image is first separated
into objects (segmentation)• Descriptors are created for each object• The user can search for a specific object
contained in images
Digital Days 29/6/2001
ISTORAMA architecture
Server
World Wide Web
Data BaseJDBC
Java Data Base Connection
User
PHP
Crawler - Spider
Indexing - Retrieval Algorithms
Digital Days 29/6/2001
The K-Means with Connectivity Constraint Algorithm (KMCC) I
• Based on K-Means algorithm• K-Means does not take into account spatial
information• In KMCC, the spatial proximity of each region is
also taken into account by defining a new spatial center and by integrating the K-Means with a component labeling procedure
• Automatic correction of the number of regions KK
Digital Days 29/6/2001
The K-Means with Connectivity Constraint Algorithm (KMCC) II
• Step1 K-Means is performed • Step2 Spatial centers are calculated
• Step3 Generalised distance
• Step 4 Component labeling LL connected regions
kCI
k
k
Sk
I AAkD
CSpCIpIp
22
1)(),(
kCS
Digital Days 29/6/2001
The K-Means with Connectivity Constraint Algorithm (KMCC) III
Digital Days 29/6/2001
Object descriptors
• Color, texture and spatial characteristics
• Color: histogram, 8 bins
• Spatial: (centroid),
• Shape: area, eccentricity
where λ1, λ2 are the two first eigenvalues
kCS
2
11
Digital Days 29/6/2001
Experimental Results (Synthetic)
Digital Days 29/6/2001
Experimental Results (Synthetic)
Digital Days 29/6/2001
Experimental Results
Digital Days 29/6/2001
Experimental Results (Claire)
Facial region
Moving object
Original sequenceFrames 1-
10
Segmentation
Digital Days 29/6/2001
Experimental results (Claire)
Digital Days 29/6/2001
Experimental results (table-tennis)
Original sequenceFrames 1-10
Digital Days 29/6/2001
Experimental results (table-tennis)
Segmentation
Moving objects
Digital Days 29/6/2001
Experimental results (Akiyo+Foreman)
Facial region
Facial region
Original sequenceFrames 1-
10
Original sequenceFrames 1-
10
Digital Days 29/6/2001
Conclusions
• K-means with spatial proximity algorithm• Multiple features segmentation• Higher order segmentation• Correspondence of objects between consequent
frames• Max-min criterion for automatic regularisation
parameters
Digital Days 29/6/2001
Future work
• Use of texture
• Indexing of video
• Integration with text descriptors
Digital Days 29/6/2001
• Triangular meshes of high quality are used in:
• Computer Aided Design • 3D representation of objects
(e.g. archaeological artifacts)• Animation and visual simulation• Entertainment (computer games)• Digital Terrain Modelling
Introduction
Digital Days 29/6/2001
Object/model-based coding
Digital Days 29/6/2001
Object/model-based coding
Digital Days 29/6/2001
Object/model-based coding
Digital Days 29/6/2001
Compression of finely detailed surfaces is necessary for:
• computation
• storage
• transmission
• display efficiency
Adaptive triangulation
Digital Days 29/6/2001
• Early, coarse approximations are refined though additional bits
Progressive transmission
Digital Days 29/6/2001
• Vertices removal and retriangulation [Schroeder] [Cohen]
• General mesh optimization process/function [Hoppe]
• Multiresolution analysis (MRA) [Lounsbery]
• Wavelets [Schroeder] [Gross]
• Progressive transmission [Schroeder] [Hoppe]
• Generalized triangle mesh representation [Deering]
Background
Digital Days 29/6/2001
Properties of the algorithm
• Efficient compression of the wireframe information• Simplification of the wireframe by adaptive
triangulation• Progressive transmission of the wireframe
information• Prioritised transmission of the wireframe• Straightforward correspondence between
successive scales
Digital Days 29/6/2001
Input surfaces
• Surface represented as a parametric function
in the parametric space
• determined by the position of a set of control points or nodes
• It allows for arbitrary, possibly closed wire-frame surfaces to be defined.
TvuzvuyvuxvuP ),(),,(),,(),(
2R
Tklklkl zyxlkr ,,),(
Digital Days 29/6/2001
Input surfaces
• The filters are applied to the 2D parametric representation of the surface as though it were a 2D image with intensity equal to
• Such surfaces include also:• depth images estimated from stereo pairs and• every surface that is homomorphic to a plane,
cylinder or torus
),( vuP
),( vuz
Digital Days 29/6/2001
Block diagram of the proposed procedure
Digital Days 29/6/2001
Reduced pyramid with quincunx sampling matrix
Digital Days 29/6/2001
Corresponding triangulation
Digital Days 29/6/2001
Optimal filters
• Optimal filters are determined by their Fourier transform:
• where is the power spectral density.
• Alternatively may be determined by the equation:
1,,1,0,1
021
Nr
eeeG M
iqj
rM
jrj
ri
T Mw
ww
Mk)(ig
wjr e
tktMMkpMtk
,)()(r
irir RgR
Digital Days 29/6/2001
Optimum bit allocation
• bits/vertex is assumed to be transmitted
• bits/vertex are allocated to each level using
• is the sum of error variances
BB rr 2
2
B
rB
2
r
ir
0
22
Digital Days 29/6/2001
Error prioritization
• The prediction errors corresponding to all predicted vertices are calculated and sorted with the vertices corresponding to higher errors being put first on the list
Higher Errors
Lower Errors
Digital Days 29/6/2001
Entropy estimation
• Entropy coding is used• The number of bits needed for error transmission
is the entropy of the errors • Using the quincunx sampling geometry at the
receiver, there is no need to transmit the exact co-ordinates of the position of each transmitted vertex
• The final cost of the transmission is the sum of the error entropy and the position entropy
Digital Days 29/6/2001
Adaptive Triangulation Procedure
• Synthesis stage of the QMVINT pyramid
• The vertex along with the vertices used to predict it are added to the mesh
• Handling of cracks
• Triangulation of the next vertex
)()(ˆ)( )()()( rk
rk
rk PerrPIPI
)(rkP
Digital Days 29/6/2001
Adaptive triangulation procedure
Digital Days 29/6/2001
Adaptive triangulation procedure
Digital Days 29/6/2001
Experimental results
• Original dense depth map and surface of the “Venus” data
Digital Days 29/6/2001
Experimental results
• 2569 vertices and 4006 triangles at level 2 MSE = 1.30
Digital Days 29/6/2001
Experimental results
• 7661 vertices and 11135 triangles at level 1 MSE = 1.30
Digital Days 29/6/2001
Experimental results
• 11416 vertices and 15827 triangles at level 0 MSE = 0.12
Digital Days 29/6/2001
Experimental results
Digital Days 29/6/2001
Conclusions
• Hierarchical representation of 3D surfaces using 3D adaptive triangular wireframes
• The variance of the error transmitted is minimised and therefore results to optimal compression of the wireframe information
• It produces a hierarchy where coarse meshes are as similar to their finer versions as is possible
Digital Days 29/6/2001
Conclusions
• The triangulation algorithm is integrated with a bit allocation procedure
• The number of nodes and triangles of the wireframe as well as the information needed for the transmission or storage of the wireframe are reduced simultaneously using a unified approach (QMVINT filtering)
• Precise correspondence between triangles at each level is achieved
Digital Days 29/6/2001
Future work
• Expansion and application directly to 3D surfaces
• Estimation of filters