inaoe at imageclef2007 towards annotation based image retrieval
DESCRIPTION
INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval. H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín, Manuel Montes , Eduardo Morales, Enrique Sucar, Luis Villaseñor Language Technologies Laboratory National Institute of Astrophysics, Optics and Electronics - PowerPoint PPT PresentationTRANSCRIPT
INAOE at ImageCLEF2007Towards Annotation based Image Retrieval
H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín, Manuel Montes, Eduardo Morales, Enrique Sucar, Luis Villaseñor
Language Technologies LaboratoryNational Institute of Astrophysics, Optics and Electronics
Tonantzintla, Mexico
[email protected]://ccc.inaoep.mx/~mmontesg
Overview of the talk
• Our first participation at ImageCLEF; the goal
was to build the basic infrastructure
– Some textual and mixed strategies for image retrieval
• However we could do something more…
– A Web based query expansion method, and
– An annotation based image retrieval approach
Textual and mixed strategies
• VSM IR System for textual retrieval (baseline)
• Late fusion of independent retrievers (LF)
• Intermedia feedback (IMFB)
TBIR
CBIR
FusionRelevantImages
Query
Example images
Topic statement
Query CBIR
QueryExpansion
TBIRRelevantImages
Topic statement
Example images Terms for annotations
Some new things…
• Web-based query expansion:– Original statement + top-k snippets (NQE)
– Original statement + top-l more repeated words from the top-k snippets (WQE)
• Annotation based expansion (ABE)
– Use automatic image annotation methods for obtaining text from images, then…
– Expand documents and/or queries with automatic annotations, finally…
– Apply some strategy for textual image retrieval
Basis of our idea
skypalm
clouds
sea
sandsand
grass
palm, sky, sand, grass, sea, clouds
Flamingo Beach Original name in Portuguese: “Praia
do Flamengo”; Flamingo Beach is considered as one
of the most beautiful beaches of Brazil;Flamingo Beach Original
name in Portuguese: “Praia do Flamengo”; Flamingo
Beach is considered as one of the most beautiful
beaches of Brazil;
Flamingo Beach Original name in Portuguese: “Praia
do Flamengo”; Flamingo Beach is considered as one
of the most beautiful beaches of Brazil;
• Region-level annotations are generally complementary to manual (image-level) annotations
Automatic image annotation
• Assign labels (words) to regions within segmented images
x1 x2 …….. xn Automatic imageAnnotation
method
Grass 0.6Sky 0.2Tree 0.1Ground 0.1
Grass 0.5Tree 0.3Ground 0.1Jet 0.1
Rock 0.5Church 0.2Elephant 0.2Entrance 0.1
Elephant
Grass
Sky
x1 x2 …….. xn
x1 x2 …….. xn
. . .
Annotation improvement
Improving the automatic annotation
Grass 0.6Tree 0.2rock 0.1building 0.1
People 0.4Tree 0.3Mountain 0.2Jet 0.1
Church 0.3Grass 0.3Sky 0.2Elephant 0.2
Tree 0.5Grass 0.3Sky 0.1Jet 0.1
R1 R2 R3 R4
c1 grass people tree churchc2 grass tree tree churchc.. …. …. …. ……ci rock people tree churchc.. …. …. …. ……cj tree people tree churchc256 building jet jet elephant
Idea: select the best label’s configuration, taking into account:1. The prior probabilities of each label, and2. The semantic cohesion of the entire configuration
Grass, Tree, Rock, Building, People,
Mountain, Jet, Sky, Church, Elephant
Set of labels
Some problems with the labels
2000 training annotated-regions (2%) 98000 regions to annotate (98%) Imbalanced training set Limited vocabulary
Annotation based query expansion
sand boats sky tree people water sand sky tree buildings sky water tree
accommodation with swimming pool
accommodation with swimming pool +sand boats sky tree people water buildings +
three given images
The volcano Tungurahua
Baños, EcuadorMarch 2002
sand clouds sky mountain
The surroundings of the Valle Francés Torres del
Paine National Park, Chile March 2002
furniture grasspeople clouds
Annotation based document expansion
Experimental results
Top ranked runs for each configuration considered.
Visual-English run
• No textual query was used, but at the end the recovery was done based on textual data.
• It combines intermedia feedback and our annotation based expansion technique.
CBIR
TBIRRelevantImages
Exampleimages
Terms for manual annotations
AutomaticAnnotation
Automaticannotations
Textual vs. mixed strategies
Initial conclusions
• Intermedia feedback is an effective way for mixing visual and textual information
• Methods based on web-query expansion showed better performance
• Anotation based expansion is a promising way for expanding text using image’s visual content
• Annotations can be useful for image retrieval, though several issues should be addressed
Our current work
• Work on the improvement of automatic image annotation methods
• Investigate different (better) ways for measuring the semantic cohesion between labels and manual annotations
• Use such semantic cohesion estimates for improving image retrieval from annotated collections
Thanks for your attention
Language Technologies LaboratoryNational Institute of Astrophysics, Optics and Electronics
Tonantzintla, México
Manuel Montes y Gó[email protected]
http://ccc.inaoep.mx/~mmontesg