e.g.m. petrakisvisual content1 retrieval of visual content images comprise the vast majority of...
Post on 22-Dec-2015
216 views
TRANSCRIPT
E.G.M. Petrakis Visual Content 1
Retrieval of Visual Content
Images comprise the vast majority of data in many application domains Remote sensing (NASA, 1 terabyte per
day)AstronomyGeographic Information Systems (GIS)Medicine (CT, MRI, etc.)Criminal investigationTrademark authentication
E.G.M. Petrakis Visual Content 2
Images in Multimedia Systems
Images co-exist with other types of data in Multimedia Documentstextattributevideosound
E.G.M. Petrakis Visual Content 3
Content-Based Image Retrieval
Descriptions of image content are extracted and stored
Manually: mainly text descriptions DifficultSubjective
Automatically: features from contentComputationally expensiveInexactDomain specific
E.G.M. Petrakis Visual Content 4
System Architecture
E.G.M. Petrakis Visual Content 5
Design Issues
Feature Extraction (functions)Feature SelectionOrganization of stored information, file
structures, indexingSearch and retrieval strategies
Sequential / Indexed search / Query refinement
Query language: conditional / example queries
User interface design
E.G.M. Petrakis Visual Content 6
Image Descriptions
Subjective interpretation of content: means different things to different people
Different features for different applicationsColour is important of out-door image but not
for X-rays, CT, MRI etc.Motion features are sometimes important
(ultrasound)
Different systems for different applications
E.G.M. Petrakis Visual Content 7
Levels Representation
Low at pixel level (e.g., intensities, colors)
Intermediate at region level (e.g., region, shape, motion features, motion)
High – Semantic human interpretations (e.g., a class per object or image or domain concepts such as diagnosis …)
E.G.M. Petrakis Visual Content 8
Conflicting issues
Dependence on image content, computational overhead and uncertainty increases from low to high level
Selection depends on application, image type, user requirements, query types
E.G.M. Petrakis Visual Content 9
Reliability Criteria
Uniqueness Proportionality of variationRobustness against noiseInvariance under translation,
rotation, scalingComputationally efficientContent at various level of detail
E.G.M. Petrakis Visual Content 10
Generic Features
Feature vectors of intensity / colortexturespatial relationshipsmotioncombinations of the above
Two kinds of featuresglobal: computed for the entire imagelocal: computed for objects or image
parts
E.G.M. Petrakis Visual Content 11
RGB Color Space
Popular hardware oriented scheme
Colors form a unit cube r = R/(R+G+B)g = G/(R+G+B)b = B/(R+G+B)
RGB is good for acquisition and display but not for the perception of colors
E.G.M. Petrakis Visual Content 12
Munsell Color Space
Color in cylindrical coordinates Brightness: vertical
axis Hue: angular
displacement Saturation:
cylindrical radius
E.G.M. Petrakis Visual Content 13
Color Definitions
Brightness: intensity of color, average intensity over all wavelengths
Hue: proportional to the average wavelength of the color percept
Saturation: amount of white, highly saturated colors have no whitedeep red has S=1pinks have S=0
E.G.M. Petrakis Visual Content 14
HSV Color Space
Value
Hue
Saturation
H = undefined for S = 0H = 360 – H if B/V > G/V
)++(31
= BGRV
))(()(2
212
cosBGBRGR
BGRH
),,min(1= ++3 BGRS BGR-
E.G.M. Petrakis Visual Content 15
Color in Retrievals
Color Histograms are very commonSimple to compute and compareFor the entire image or for image parts 3D histogram on RGB or HSV space (224 bins!)1D histogram over the 3 primaries (256 bins)
Use HSV histograms: changes in lighting and viewing angles may cause major variations in RGB histograms
Invariant under translation, rotation, viewing angle and scaling
E.G.M. Petrakis Visual Content 16
A.Del Bimbo 99
1D histogram
E.G.M. Petrakis Visual Content 17
Histogram Comparison
Histogram intersectionQ, I: histograms of a
query and database imageN: histogram bins3D (RGB, HSV)
intersection is defined accordingly
N
i i
N
i ii
Q
QIQIS
1
1),min(
),(
A.Del Bimbo 99
normalized intersection
E.G.M. Petrakis Visual Content 18
Reducing Complexity
Reduce number of histogram binsTransform RGB histogram to (rg,by,wb)rg = R – G, by = 2B – R – G, wb = R + G +
BIntensity wb is more coarsely sampled than
rg, by wb (8 sections), rg, by (16 sections)The resulting histogram has 2048 binsReduced sensitivity to variations of
intensity
E.G.M. Petrakis Visual Content 19
Reducing Complexity (cont,d)
Clustering detects the K most prominent colors (e.g., K-means)Histogram with K bins (e.g., K=64 or
256)Each bin is the normalized count of
pixels in the cluster
E.G.M. Petrakis Visual Content 20
Reducing Complexity (cont,d)
Recognize that only a small number of bins capture the majority of pixelsThreshold to take only the large binsSmall bins are likely to be noisy bins
thus distorting the intersectionDoes not degrade the performance
E.G.M. Petrakis Visual Content 21
Distance Function
Certain pairs of bins correspond to perceptually similar colorsIn intersection all bins are compared
independently of each otherDefine new Distance function:
A=(aij) represents bin proximity
aij based on proximity in the L*u*v space
K
i
K
j jijiijt yxyxaQIAQIQID
1 1
2 ))(()()(),(
E.G.M. Petrakis Visual Content 22
A.Del Bimbo 99
L*u*v color space
E.G.M. Petrakis Visual Content 23
Color Indexing
Color (feature) vector: histogram Problems:
K is large (K=64 or 256)Quadratic complexity of matchingSAMs assume independent attributes
Solution: GEMINIMap to low dimensionality feature space Lower bound distance: Df(I,Q) <= D(I,Q)
E.G.M. Petrakis Visual Content 24
Definition of Df(I,Q)
Take some average color value on color space (e.g., R,G,B)average color of image: (Ravg,Gavg,Bavg)=
and
P
i
P
i
P
i
avgavgavg
pBPpGPpRP
BGR
111)(/1),(/1),(/1
),,(
3
1
2)-()-()-(),(i ii
tf yxyxyxQID
E.G.M. Petrakis Visual Content 25
GEMINI Approach
Indexing in the 3D color space Df < D(I,Q): see QBIC paper for proof
Map query Q to the same 3D space Search the feature space Clean-up answer set to eliminate
false drops
E.G.M. Petrakis Visual Content 26
Texture
Repeative patterns of local variations of intensity
Structural: identify placement rules of structural primitives the less effective approach
Statistical: characterize spatial distribution of intensity in terms of measurementsHaralick, Tamura features etc.
E.G.M. Petrakis Visual Content 27
Texture Examples
Ballard and Brown 84
E.G.M. Petrakis Visual Content 28
Structural Texture
Ballard and Brown 84
E.G.M. Petrakis Visual Content 29
Statistical Texture
Ballard and Brown 84
E.G.M. Petrakis Visual Content 30
Haralick Features [Haralick 73]
Set of 4 features characterizing the intensity transitions of neighboring pixels in various directions using Gray-Tone Spatial-Dependence (GTSD)
arraysOne GTSD for each pixel neighborhood Neighborhood: pixels in direction θ and
distance d
E.G.M. Petrakis Visual Content 31
GTSD Array Pd,θ[i,j]
Counts pixel pairs in distance d having gray levels i, j in direction θOne GTSD for θ=(00, 450,900, 1350) and d=(1,2,..)Intensity in range [0,k-1]: Pd,θ[i,j] is a k x k matrix
2 1 2 0 1
0 2 1 1 2
0 1 2 2 0
1 2 2 0 1
2 0 1 0 1
i
j
0 2 2
2 1 2
2 3 2
1
16
P[i,j]d = 1
0 1 2
0
1
2
E.G.M. Petrakis Visual Content 32
Computing Pd,θ[i,j]
Count all pairs of pixels in which the first pixel has value i and its matching pair displaced by d=1 in θ = 450 or 1350 direction has value j
Enter this count in the (i,j) position of Pd,θ[i,j]E.g., there are 3 pairs [2,1], then P[2,1] = 3Pd,θ[i,j] is not symmetric: Pd,θ[i,j] < > Pd,θ[j,i]
Normalize Pd,θ[i,j] by the total number of pairs
Pd,θ[i,j]: probability mass function
E.G.M. Petrakis Visual Content 33
Texture Features
1) Angular Second Moment (ASM):
Small values for non homogeneous regions
2) Contrast: Large values for many large transitions
or for many transitions
1
0
1
0
21 ),(
k
i
k
jjipf
1
0
1
0
22 )](,[
k
i
k
jjijipf
E.G.M. Petrakis Visual Content 34
Texture Features (cont,d)
3) Correlation:
Frequency of intensity transitions
1
0y
1
0j
2y
1
0
1
1x
1
0
2x
1
0
1
0
1
03
],[ p ,][)( σ ,][
],[ p , ][)( σ ,][
],[
k
i
k
yy
k
j yy
k
j
k
i xx
k
i xx
yx
yx
k
i
k
j
jipjpjjjp
jipipiiip
jiijpf
E.G.M. Petrakis Visual Content 35
Texture Features (cont,d)
4) Entropy:
High values for uniform p[i,j] i.e., no preferred gray-level (no texture)
A vector for each Tθ,d=(f1,f2,f3,f4) or A vector for every θ, d taking all Tθ,d in a
sequence Correlated features: apply K-L to de-
correlate and to reduce dimensionality
1
0
1
04 ],[log],[k
i
k
jjipjipf
E.G.M. Petrakis Visual Content 36
Shape
Assume that objects are extracted Requires image segmentationDifficult problem
Criteria for reliable shape recognitionUniqueness of representationRobustness against noise and distortionProportionality of variationInvariance under scale, rotation and translationEfficiency of computationOcclusion: handle partially visible shapes
E.G.M. Petrakis Visual Content 37
Shape Matching Methods
Two categories of methods based on: Regions: represent and match
properties of regionsContours: represent and match
properties of boundariesTechniques: local/global, model
based, fuzzy, statistical, neural networks
E.G.M. Petrakis Visual Content 38
Input/Output
For any two shapes and compute:Their distanceThe correspondences between similar parts
Petrakis 02
E.G.M. Petrakis Visual Content 39
Moment Invariants
An object is represented by its binary image
A set of 7 features can be defined based on central moments
00
01
00
10
),( m
my ,
m
mxyxm
Ryx
qppq
R
Ryx
qppq yyxx
),(
0,1,2...qp, ),)((
E.G.M. Petrakis Visual Content 40
Central Moments [Hu 62]
Invariant to translation and rotation Use ηpq=μpq/μγ
00 where γ=(p+q)/2 + 1 for p+q=2,3… instead of μ’s in the above formulas to achieve scale invariance
E.G.M. Petrakis Visual Content 41
More Shape Methods Moments can also be defined on the closed
bounding contours of objects [Gupta and Shinath 87]
Moments can also be defined for open curves [Koch and Kashyap 89]
Methods based on the Fourier Transform of the bounding contour have also been used [Wallace and Wintz 80, Rauber and Steiger 92]
More efficient methods has also been proposed [Petrakis, Diplaros and Milios 2002]. Examines many of the above methods based on Fourier and Moments and shows many experiments and comparisons
E.G.M. Petrakis Visual Content 42
Spatial Relationships
Find images showing similar objects in similar spatial relationships find X-rays similar to Smith’s examinationfind images showing a tree close to a houseone of the two images may contain extra objects
Q I Petrakis02
E.G.M. Petrakis Visual Content 43
Methods
Two main categories of methods Spatial projections (2D strings and
variants like 2D C strings, Expanded 2D strings etc).
Attributed Relational Graphs (ARGs)Image distance is defined
accordinglyEditing distance on ARGs2D string matching
E.G.M. Petrakis Visual Content 44
Image Segmentation
All methods assume segmented images image are segmented manually or semi-
manuallyimage segmentation is a difficult problem
Petrakis02
E.G.M. Petrakis Visual Content 45
Image Features
Individual objects: 5-dimensional vectorsSize: number of pixels in a regionPerimeter: length of bounding contourRoundness: ratio of smallest/largest second
momentorientation: angle with x direction (sin,cos)
Spatial Relationships: 4-dimensional vectorsPosition: inside or outsideDistance: minimum distance of contoursOrientation: angle with x (sin,cos) of c.g.’s
E.G.M. Petrakis Visual Content 46
Attributes Relational Graphs (ARGs)
Objects are represented by nodes
Relationships are represented by arcs
Nodes and arcs are labeled by feature vectors
Matching: ARG editing distance, Hungarian [Petrakis 02]
Petrakis02
E.G.M. Petrakis Visual Content 47
ARG Editing Distance
Matching: sequence of edit operations that transform a query Q to an image IEdit operation: node or arc insertion,
deletion or substitution
F combines the costs of edit operationsf is the cost of an edit operation defined
as a vector distance
)(),...(),(min
))((min),(
11)(
)'(
fffF
ISFIQD
kkIS
GS
E.G.M. Petrakis Visual Content 48
Matching Algorithm [Messmer95]
Find the sequence of edit operations that yield the minimum total cost
Formulated as tree search problemExpand all possible matching sequencesBranch and bound Tree node: matching of ARG nodeTree arc: matching of ARG edgesSubtree: matching of subgraphs of Q, I
E.G.M. Petrakis Visual Content 49
Query Q Model I
E.G.M. Petrakis Visual Content 50
Hungarian Method [Petrakis 02]
Matching: assignment problem
The relationships are ignored
F: cost of a mappingC(i,F(i)): vector
distance
n
iFF iFiCIQD1
))(,(min),(
E.G.M. Petrakis Visual Content 51
2D String [Chang 87]
2D string: projections of c.g.’s along x and y Each object is represented by a name or class
Matching: string matching (type 0,1 and 2)
E.G.M. Petrakis Visual Content 52
Discussion [Petrakis 02] The ARG editing distance is the most
accurate method followed by Hungarian and 2D strings
2D strings is the faster method followed by Hungarian and ARG distance
Speed and Accuracy are traded-off: the most accurate a method the slower it is
Indexing: Petrakis 2002, Petrakis & Faloutsos 97 (ARGs), Petrakis 93 (2D strings)
http://www.ced.tuc.gr/~petrakis/publications/publications.htm
E.G.M. Petrakis Visual Content 53
Image Segmentation
All methods assumed segmented imagesSegmentation is the process of partitioning
an image into groups of connected pixels (regions) with similar properties Gray levelsColors TexturesMotion characteristics (motion vectors)Edge continuity
E.G.M. Petrakis Visual Content 54
Segmentation Methods
Two approaches Region segmentationEdge segmentation
Regions may correspond to objectsNot always perfect (noise, bad
illumination, 3D world etc.)Further reading: "Machine Vision'', R.
Jain, R. Kasturi, B. G. Schunck, Mc Graw-Hill, 1995
E.G.M. Petrakis Visual Content 55
Region Segmentation
Converts a gray-level image into a binary one by applying carefully selected thresholds on intensity histograms
The image is partitioned into two setsBlack pixels: objects White pixels: background
E.G.M. Petrakis Visual Content 56
Histogram Thresholding
The threshold distinguishes the objects from the background
The objects have similar gray-level values
E.G.M. Petrakis Visual Content 57
Thresholding
Find thresholds automatically by analyzing the gray value distribution (histogram) of the image
Objects are dark against a light backgroundTheir gray-value distributions can be separated
putting thresholds between them
Automatic thresholding is based on peackiness and valleyness measurements at each point of the histogram
E.G.M. Petrakis Visual Content 58
Further Reading C. Faloutsos et.al. “Efficient and Effective Querying by Image Content
”, Journal of Intelligent Information Systems, Vol. 3, No. ¾, pp. 231-262, 1994
M. Flicknet et.al. “Query by Image and Video Content: the QBIC Systems”, IEEE Computer, Vol. 28, No. 9, pp. 13-32, Sept. 1995
R.C.Veltkamp and M.Tanase “Content-Based Image Retrieval Systems: A Survey”, TR UU-CS-2000-34, Utrecht University, March 2001
A.W.M.Smeulders et.al., “Content Based Image Retrieval at the End of the Early years”, IEEE Transactions on PAMI, 22(12): 1349-1380, 2000
R.Schettini, et al. “A Survey on Methods for Colour Image Indexing and Retrieval in Image Databases” , in: R.Luo and L.MacDonald (Eds.), Color Imaging Science: Exploiting Digital Media, John Wiley, 2001
M. Swain, D.H.Ballard, “Color Indexing”, Intern. Journ. of Comp. Vision, Vol. 7, No. 1, pp. 11-32, 1991
D. Androutsos et.al. “A Novel Vector-Based Approach to Color Image Retrieval Using a Vector Angular-Based Distance Measure”, Comp. Vision and Image Understanding, Vol. 75, No. ½, July/Aug. 1999, pp. 46-58.
E.G.M. Petrakis Visual Content 59
References J.R.Smith, S-F.Chang, “Tools and Techniques for Color Image Retrieval”,
IS&T/SPIE Proc., Vol. 2670, Storage and Retrieval for Image and Video Databases IV
R.M. Haralick, K. Shanmungam, I. Dinstein “Textural Features for Image Classification”, IEEE Trans. on Systems Man and Cybernetics, 1973, pp. 610-621.
E.G.M. Petrakis, A. Diplaros and E. Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Trans. on PAMI, Vol. 24, No. 11, Nov. 2002, pp. 1-16.
M.-K. Hu. Visual Pattern Recogn. by Moment Invariants. IRE Trans. on Info. Theory, IT-8:179–187, 1962.
T. P. Wallace and P. A. Wintz. An Efficient Three-Dimensional Aircraft Recognition Algorithm Using Normalized Fourier Descriptors. Computer Graphics and Image Processing, 13:99–126, 1980.
T.W. Rauber and A.S. Steiger-Carcao, “Shape Description by UNL Fourier Features – An Application to Handwritten Character Recognition, 11th IAPR Intern. Conf. on Pattern Recogn., 30.Aug.-3.Sept. 1992, The Hague, The Netherlands (click here for implementation).
E.G.M. Petrakis Visual Content 60
References L. Gupta and M.D. Shrinath, “Contour Sequence Moments for the
Classification of Closed Planar Shapes”, Pattern Recognition, Vol. 20, No. 3, pp. 267-272, 1987
M.W.Koch and R.L.Kashyap, “Matching Polygon Fragments”, Pattern Recognition Letters, No. 10, pp. 297-308,1989.
Euripides G.M. Petrakis, Aristeidis Diplaros and Evangelos Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, November 2002, pp. 1-16.
Euripides G.M. Petrakis, Aristeidis Diplaros and Evangelos Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, November 2002, pp. 1-16.
Euripides G.M. Petrakis: "Fast Retrieval by Spatial Structure in Image DataBases", Journal of Visual Languages and Computing, Vol. 13, No. 5, October 2002, pp. 545-569.
Euripides G.M. Petrakis: "Design and Evaluation of Spatial Similarity Approaches for Image Retrieval", Image and Vision Computing, January 2002, Number 1, Volume 20, pp. 59-76.
Euripides G.M. Petrakis and Christos Faloutsos: "Similarity Searching in Medical Image Databases", IEEE Transactions on Knowledge and Data Engineering, Vol. 9, No. 3, pp. 435-447, May/June 1997.