workshop on semantic knowledge in computer vision, iccv 2005 symbol grounding for semantic image...
TRANSCRIPT
Workshop on Semantic Knowledge in Computer Vision, ICCV 2005
Symbol Grounding for Semantic Image Interpretation:
From Image Data to Semantics
Céline Hudelot, Monique Thonnat and Nicolas Maillot
INRIA Sophia Antipolis, FRANCE
16/10/2005 SKCV 2/24
Outline
Introduction Symbol Grounding Problem Ontology-based Communication Learning Approach Knowledge-based Approach A Symbol Grounding Engine Conclusion
16/10/2005 SKCV 3/24
Introduction
Problem: What does it means to perform
semantic image interpretation ?
What does it means to associate semantics to a particular image ?
16/10/2005 SKCV 4/24
Introduction
Different interpretations are possible
Image semantics is not inside the imageImage interpretation depends on a priori knowledge and on the context
• A white object on a green background
• An insect• An infection of white
flies on a rose leaf
16/10/2005 SKCV 5/24
Introduction
Three abstraction levels of data Vision [Marr,82], Cognitive Science [Gardenfors,2000]
Semantic level
Image levelRegion 1:Area : 105compactness :0.9Circularity : 0.85HSV (0.05,0.2, 0.6) ...
APPLE
FRUIT
PEAR ORANGE
PLATE_OF_FRUITSComposition link
Specialization link
STEM
PEACH
Visual level
A circular shape, orange hue and regular granulated texture
16/10/2005 SKCV 6/24
Introduction
Three sub-problems: Image processing : extraction of numerical
image data Region 1:Area : 105compactness :0.9Circularity : 0.85HSV (0.05,0.2, 0.6) ...
SEGMENTATIONFEATUREEXTRACTION
Region 1:Area : 105compactness :0.9Circularity : 0.85HSV (0.05,0.2, 0.6) ...
Orange Fruit :Has for shape : circularHas for hue: orangeHas for texture : granulated
Symbol grounding : mapping between image data and high level representations of semantic concepts
Symbol grounding
Semantic interpretation : reasoning at the high level
16/10/2005 SKCV 7/24
The Symbol Grounding Problem
Definition: Problem of the mapping between image
data and semantic data
Objective Propose generic tools to solve the symbol grounding problem as a problem as such
Area : 105compactness :0.9Circularity : 0.85HSV (0.05,0.2, 0.6) ...
The Orange Fruit
16/10/2005 SKCV 8/24
The Symbol Grounding Problem
Proposed Approach An independent intermediate level called
visual level between the semantic level and the image level
Two ontologies to make easier the communication between the different levels
Visual concept ontology Image processing ontology
A cognitive vision approach involving a priori knowledge and machine learning
16/10/2005 SKCV 9/24
The Symbol Grounding Problem
Proposed ApproachSemantic level
Image level
Region 1:Area : 105compactness :0.9Circularity : 0.85HSV (0.05,0.2, 0.6) ...
Visual level
A circular shape, orange hue and regular granulated texture
Orange Visual concept ontology
Image processing ontology
Symbol grounding problem : matching image data with combination of visual concepts
16/10/2005 SKCV 10/24
The Symbol Grounding Problem
Proposed Approach Build the correspondence links between
images features and visual concepts Learning approach : the correspondence links
are learned from images samples A priori knowledge based approach: links are
built explicitly and stored in a knowledge base
A symbol grounding engine uses these links to perform the matching
16/10/2005 SKCV 11/24
Ontology Based Communication
A visual concept ontology [Maillot et al. 03] Experts often use and share a generic visual
vocabulary to describe their domain A hierarchy of three kinds of 2D visual concepts
Spatial Concepts (64 concepts) Shape, Size: circular, elongated,… Spatial Structure : network of, ring of,… Binary spatial relations : near of, connected to, left of
Color Concepts (37 concepts) : red, light, vivid (ISCC-NBS lexicon)
Texture Concepts (14 concepts) : granulated, regular (cognitive studies [Bhushan,97])
Application independent A basis for further extensions
16/10/2005 SKCV 12/24
Ontology Based Communication
Why a Visual Concept Ontology ? To guide and constrain the semantic knowledge
acquisition
Reduce the semantic gap : a shared representation of image content at an intermediate level
Communication between the semantic level and the visual level
Domain Expert
Images Samples
Visual Concept Ontology
Knowledge Acquisition Knowledge Base
Manually Segmented
and Annotated
Images
16/10/2005 SKCV 13/24
Ontology Based Communication
An Image Processing Ontology Domain of discourse of image processing: set of
generic terms to describe images and image processing results
“Images have an ontological description of their own” Hierarchical set of :
Image entity concepts : region, edge, graph …(11 concepts)
Image feature concepts : eccentricity, RGB values, … (167 concepts)
Image processing functionalities :object_extraction, feature_extraction,… (5 generic functionalities)
Communication level between the image level and the visual level
Not complete, a basis for further extensions
16/10/2005 SKCV 14/24
Supervised Learning Approach
Goal: Training a set of detectors (e.g. Multi Layer perceptrons, SVM) to the detection of visual concepts Each visual concept C is associated to a set of
image features FC
Only visual concepts used during the semantic knowledge acquisition phase are learned
Positive and negative
samples of each visual
concept
VisualConcept
Detectors
Feature
SelectionTraining
Feature
Extraction
16/10/2005 SKCV 15/24
Supervised Learning Approach
Example : learning of the visual concept granulated texture
Granulated TextureDetector
LDA NN
Positive and negative samples of visual concept
Granulated Texture
Gabor
Filter
circular shape orange hue granulated texture
Manually segmented and annotated images
16/10/2005 SKCV 16/24
Supervised Learning Approach
Reduce the learning problem by addressing it at an intermediate level of semantics
Automatic building of the symbol grounding link between visual concepts C and image features F
Does not learn spatial structure and spatial relations
Dependent on the learning base : a large amount of image samples is needed
16/10/2005 SKCV 17/24
A Priori Knowledge Base Approach
Explicit representation with frames: Visual concepts (symbolic data): description of
visual concepts C and of their grounding link with image features F
Image data concepts (image data): primitives (ridge, region, edge), features (area, eccentricity) organized in feature sets
Spatial relations : topology (RCC8), distance and orientation
Explicit representation with rules: Object extraction criteria: to constrain image
processing Spatial deduction criteria: to infer spatial
relations
16/10/2005 SKCV 18/24
A Priori Knowledge Based Approach
Visual concept : simple examplesVisualConcept{name Circular_SurfaceSuper Concept Elliptical_SurfaceGrounding LinkSymbol name eccentricityComment ratio of the length of the longest chord to the longest chord perpendicular to itLinguistic-values [ high very_high]FuzzySet Fhigh ={0.57, 0.62, 0.76, 0.84}Fvery_high ={0.76, 0.84, 1, 1}Domain [0 1]Symbol name compactnessLinguistic-values [ high very_high] …}
VisualConcept{name OrangeSuper Concept HueGrounding LinkFloat name H_valueDomain [0.0 0.1]Float name L_valueDomain [0.5 1.0]
}
16/10/2005 SKCV 19/24
A Priori Knowledge Base Approach
Explicit representation of spatial relations [Le Ber, 98] : distance, orientation, topology (Binary, 2D)
Spatial Relation{name Externally_ConnectedSuper Relation DiscreteInverse Externally_ConnectedComplement NoneSymmetry trueConditionsIntersection(Interior(O1), Interior(O2))=ØIntersection(Boundary(O1),Boundary(O2))!=ØObjects_In_RelationVisualObject name O1VisualObject name O2}
Spatial Relation{name Near_ofSuper Relation DistanceRelationInverse Near_OfComplement Far_FromSymmetry trueFloat name distance_thresholdConditionsDistance(O1,O2) < distance_thresholdObjects_In_RelationVisualObject name O1VisualObject name O2}
16/10/2005 SKCV 20/24
A Priori Knowledge Base Approach
Object extraction criteria: how to constrain image processing (using visual concepts and spatial relations)
Example
Spatial deduction criteria : how to infer spatial relations from other ones
Example:
Rule { Let c a visual content context and O a visual objectIf O.geometry is a Open Curve and O.width is {Thin, Very Thin} then c.ImageEntityType:=Curvilinear Structure }
Rule { Let O1, O2, O3 three visual objectsIf NTTP(O1, O2) is true and Left_Of(O2,O3) is true then Left_Of(O1,O3) is true}
O2
O1O3
16/10/2005 SKCV 21/24
A Priori Knowledge Base Approach
Reduce the learning problem by addressing it at an intermediate level of semantics
No need of image samples Spatial relations are explicit Manual building of the symbol
grounding links between visual concepts C and image features F
Difficult to express some criteria for texture
16/10/2005 SKCV 22/24
Symbol Grounding Engine
Symbol Grounding (Symbols, Image) Image processing request building using object
extraction criteria Primitive selection (region, ridge,…) Feature extraction
Matching between image processing results (image features F) and symbolic data (visual concepts C)
Fuzzy Matching using explicit knowledge (Frames) OR, Matching using the detectors obtained during the
learning Spatial Reasoning for multiple objects management
using spatial deduction criteria and spatial relations
16/10/2005 SKCV 23/24
Conclusion
The two methods have been tested on real world applications A priori knowledge based approach :
Automatic early diagnosis of rose disease [Hudelot et al 03]
Supervised learning approach : Application on aircraft/cars retrieval [Maillot et al 05]
Two complementary methods The symbol grounding link is difficult to build
explicitly by a human expert in vision (e.g. texture concepts)
A large amount of data (image examples) is not available for all the applications
16/10/2005 SKCV 24/24
Conclusion
Original Symbol Grounding Approach: Ontology-based Approach
Visual concept ontology and Image processing ontology
Independence between application domain semantics and image processing library
Symbol grounding link Either learned from samples or a priori knowledge
Future works Learning for spatial relations Extension of the visual concept ontology
Temporal concepts
16/10/2005 SKCV 25/24
Symbol Grounding for Semantic Image Interpretation: From Image Data to Semantics
Any Questions??
16/10/2005 SKCV 26/24
The Symbol Grounding Problem
Related Works Knowledge based Vision:
Not often considered as a problem as such Encapsulated in the semantic level
Intermediate Symbolic Representation [Brolio,89]
VISIONS system [Hanson,78] Database management technology
Conceptual Spaces [Chella, 1997] Conceptual space = metric space which dimensions
are entity qualities Natural concepts = convex regions in the conceptual
space
16/10/2005 SKCV 27/24
The Symbol Grounding Problem
Related Works Artificial intelligence : the Symbol grounding
problem [Harnad, 90] Robotics community: the Anchoring problem
« Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99]
Image retrieval community : the semantic gap
Use of ontological engineering: object ontology [Mezaris, 04] , visual ontology [Mao,98], ontology for language based querying [Town, 04]
16/10/2005 SKCV 28/24
Ontology Based Communication
Ontology : set of concepts and relations useful to describe a domain
“A formal, explicit specification of a shared conceptualization” [Gruber, 93] Conceptualization : abstract relevant model of a
phenomenon Explicit : the meaning of the concepts is defined
explicitly Formal : machine readable Shared : consensual knowledge accepted by a
group