ontology based object learning and recognition

Ontology Based Object Learning and Recognition

PhD Defence

14/12/2005

Supervised by Monique Thonnat

Nicolas MAILLOTOrion team

INRIA Sophia Antipolis

N. Maillot 2/54

• Introduction• State of the Art• Knowledge Acquisition• Visual Concept Learning• Object Categorization• Results• Conclusion

Outline

N. Maillot 3/54

Introduction

• Context: Semantic image interpretation• Goal: Object recognition • More precisely: object categorization (i.e. finding the category of an object) and not object identification (i.e. recognition of an individual)

• Approach: Cognitive vision techniques [ECVision Roadmap 04]

• Mixing knowledge representation, machine learning, image processing and reasoning techniques

N. Maillot 4/54

Introduction: Semantic Image Interpretation

Oslo Accords (1993)

• Semantics is not inside the image:

handshake agreement

Need of a priori knowledge in international politics

N. Maillot 5/54

Introduction: object categorization

• Assigning a category (e.g. Aircraft, Galaxy) to a region of the image

• Categories are discrete entities characterized by properties shared by their members

Aircraft

N. Maillot 6/54

Introduction: Goal

• Issues:• Knowledge acquisition• Semantic gap• Use of acquired knowledge for performing object categorization

• Goal: Enabling experts to build object categorization systems dedicated to his/her domain of interest (e.g. biology)

• Restricted scope:• One main object per image • Need of a well-defined expertise

N. Maillot 7/54

Introduction: Proposed Approach

• Decomposition of the object categorization problem in three levels of abstraction:

High-Level Interpretation

Mapping

Image Processing

Domain knowledge

Knowledge about the mapping between domain knowledge and image processing knowledge

N. Maillot 8/54

Introduction: Proposed Approach

• Use of ontological engineering combined with machine learning techniques

Reduction of the knowledge acquisition problem and of the semantic gap

Performing categorization as experts do

N. Maillot 9/54


Outline

N. Maillot 10/54

State of the Art : Object Recognition

• [Brooks83] Object modeling by ribbons. Geometric reasoning

• [Havaldar96] use of qualitative geometric relationships (e.g. proximity, symmetry)

• [Basri96] Combination of alignment method with recognition by prototypes

• [Sangineto03] Recognition based on the shape invariants of object categories

Geometric model alignment

• Geometric Methods

N. Maillot 11/54


• Appearance-Based Methods

Implicit objects modelsUse of multiple views • [Swain and Ballard 91] Objects represented by color histograms

• [Schmid97] Local features. Introduction of a voting algorithm

• [Schiele00] Receptive field histograms for approximating the local appearance

• [Fergus03] Local features. Objects modeled as constellations of parts

N. Maillot 12/54


• Knowledge-Based Methods [Crevier97]

• [Draper89] Blackboard architecture. Schemas (frames + procedures). Hypothesis generation/verification.

• [Matsuyama90] Three expert systems. Frames + rules. Both model driven and data driven. hypotheses generation/verification.

• [Hudelot05] Cooperation between three knowledge-based systems (Frames + rules). Data management functionalities.

N. Maillot 13/54


• Summary:• Geometric Methods

+ Strong theoretical foundations- identification of individuals and not categorization- Reliable Extraction of geometric primitives is very difficult

• Appearance-Based Methods+ Effective- Need of large number of samples- Lack of explicitness

• Knowledge-Based Methods+ Explicit+ Separation between knowledge and reasoning

- Knowledge acquisition bottleneck (mapping knowledge is difficult to acquire)

N. Maillot 14/54


N. Maillot 15/54

Domain Expert

Visual Concept

Ontology

Knowledge Acquisition

Knowledge Base

Knowledge acquisition guided by a visual concept ontology (i.e geometry, texture, color) to

describe the objects of the domain.


N. Maillot 16/54


• Ontology • Definition: An explicit specification of a conceptualization [Gruber93]

• Composed of:• A set of concepts• A set of relations between concepts• A set of axioms (e.g. transitivity, reflexivity)

• Ontological Commitment [Bachimont2000] Shared reference to align with

N. Maillot 17/54


• Visual Concept Ontology• 144 concepts :

• spatial concepts (geometry, size, position, orientation)

• color concepts (hue, brightness, saturation)

• texture concepts (pattern, contrast, repartition)

Object classes are described by visual concepts

N. Maillot 18/54

Texture

Repartition Pattern

Repetitive Random Regular Oriented Granulated Coarse Complex

Visual concept ontology content: some texture concepts


Based on cognitive experiments [Bhushan et al 97]

N. Maillot 19/54

Subpart Tree

Poaceae :

• Circular Shape

• Granulated Texture

• Pink Color

Poaceae

Pollen

Pore Cytoplasm

Pore:

• Subpart of Poaceae

• Elliptic Shape

• Small Size

Domain knowledge described using visual concept ontology


N. Maillot 20/54

• Knowledge Formalization• Domain class hierarchy: from general to specialized classes

• Domain Partonomy: subparts linked to domain classes

• Class: a category (e.g. aircraft, pollen grain) described by visual concepts

• Representation by frames with slots


N. Maillot 21/54


Visual Concept Examples Numerical Features

ShapePolygonal, Straight SIFT Features [Lowe 99]

TextureGranulated, Smooth

Gabor Features [Manjunath 96]Co-Occurrence Matrices

ColorBlue, Bright, Dark

HistogramsColor Coherence Vectors [Pass96]

Each visual concept is associated with numerical features:

N. Maillot 22/54


• Importance of acquisition context• Visual description is valid for an image acquisition context

Acquisition Context

Point of View Sensor

RearView

FrontView

ProfileView

MicroscopeCamera

CCD Camera IR Camera

N. Maillot 23/54

Domain

class

hierarchy

Subparts

hierarchy

Ontology

driven

description

Image

samples

management


N. Maillot 24/54

Poaceae

Composition Link

Specialization Link

Pollen

Grain

Pori

NonAperturedPollen

Cupressaceae

Pori of Poaceae

Pori of Parietaria

Knowledge Base(18 domain classes + 17 visual concepts)

Cytoplasm OfCupressaceae

Pollen with Pori

Pollen with Pori and Colpi

Apertured

PollenParietariaOlea

Colpi Colpi of Olea


Context:

Sensor: Microscope

Magnification: 60

Dye: Fuchsin

N. Maillot 25/54

High-Level Interpretation

Mapping

Image Processing

Domain knowledge

Completely Acquired

Mapping Knowledge

Partially Acquired


• Conclusion:

N. Maillot 26/54


Talk Overview

N. Maillot 27/54

Visual Concept Learning

• Visual Concept Learning• Goal: Producing visual concept detectors• Why: Mapping knowledge is difficult to acquire

• How: Training of Support Vector Machines (SVM) with annotated samples

Granulated

Texture

Detector

GranulatedTexture

Confidence=0.8

N. Maillot 28/54

• Image Sample Segmentation and Annotation using visual concepts

• Three Approaches:• Manual approach• Use of 3-D models• Weakly-supervised approach


N. Maillot 29/54

Selection of an image sample of Poaceae object

Interactive selection of region of interest with a drawing tool

• Image Sample Segmentation and Annotation: Manual Approach

Annotation of selected region by visual

concepts:

- Pink- Large- Circular


N. Maillot 30/54


• Image Sample Segmentation and Annotation: Use of 3-D Models (meshes)

N. Maillot 31/54

Image training set

Automatic Segmentation

Feature

Extraction

Clustering

(k-means)

Cluster Visualization and Annotation

Annotated

Clusters

Visual concept Ontology


• Image Sample Segmentation and Annotation: weakly-supervised approach

N. Maillot 32/54

Automatic Segmentation

Size Computation

k-means

Small

Cluster Visualization and Annotation

• Example: clustering for visual concept category Size

Visual concept Ontology


Image Training Set

…

… …

… …Large

N. Maillot 33/54

• Learning (for each visual concept C used during knowledge acquisition)

Get

Positive

and

Negative

Samples Of

C

Visual

Concept

Detector

SVM

Training

Feature

Extraction

And Selection

Annotated

Regions


SVM based on Radial Basis Function Kernels

N. Maillot 34/54

Granulated

Texture

Detector

LDA SVM

• Example: Learn the visual concept Granulated Texture

Visual concept detectors are used to complete the mapping knowledge

Gabor

Filter

Get Positive

and Negative Samples

of Concept Granulated Texture

Annotated

Regions


N. Maillot 35/54


Talk Overview

N. Maillot 36/54

Object Categorization

• Object categorization based on:• Acquired knowledge (domain knowledge + mapping knowledge)

• Visual concept detectors• Mechanism: Hypothesis Generation/Verification


Input

Image

Class +

Visual

Description

N. Maillot 37/54

• Algorithm: Hierarchical exploration of object classesFor each class of the class hierarchy (from root

class)

1. Hypothesis generation: generation of a set of hypothetic visual concepts

2. Visual detection of the hypothetic visual concepts in the segmented image

3. Recursion on sub-parts

4. Hypothesis verification: object/class matching w.r.t. a matching threshold

5. If the class is verified then consider sub-classes


N. Maillot 38/54

• Matching (matching threshold=0.5)

Feature

Extraction

Automatic

Segmentation

Poaceae :

• Circular Shape

• Granulated Texture

• Pink Hue

Current Hypothesis:

CircularShape

Detector

GranulatedTextureDetector

Pink HueDetector

0.63

Σ


0.5

0.6

0.8

(0.5+0.6+0.8)/3

0.63>0.5 : hypothesis verified

?

N. Maillot 39/54

Object

CategorizationKnowledge

Base

Automatic

Segmentation

Feature

Extraction

Input Image

Poaceae 0.63Circular 0.5

Pink 0.8

Granulated 0.6


Visual Concept Detectors

Mapping

N. Maillot 40/54


Talk Overview

N. Maillot 41/54

Results

• Application: Semantic image indexing and retrieval

• Domain: Transport Vehicles (aircrafts, motorbikes, cars) in their environment

• Goal: Enabling Retrieval/Indexing by concept • User-friendliness• Efficiency: no need to store pre-computed feature vectors

• Issue: trade-off between semantic richness and amount of work needed to build semantic indexing and retrieval systems

N. Maillot 42/54

Results

• Semantic Indexing

ImageDatabase Object Categorization

Indexed Images

Use of categorization results as index for images

Indexing time: 1 sec for a 600x400 image on a Intel Pentium IV 3.06Ghz

N. Maillot 43/54

Results

• Query by concept (opposed to query by example):

Indexed Images

Semantic Query:Object Class /

Object Description

• Example of semantic queries: “Aircraft ”, “Gray Aircraft and Blue Sky ”

Retrieved Images

Retrieval

N. Maillot 44/54

Results

• [Fauqueur03] Retrieval/Indexing based on region templates

• [Town04] Supervised learning used for mapping image data to a domain ontology

• [Mezaris04] Querying based on an object ontology (color, position, size, shape). Machine learning and user feedback are used for improving system efficiency

No approach combines weak supervision with a rich high-level knowledge layer

N. Maillot 45/54

Results

Composition Link

Specialization Link

Outdoor

Scene

Transport

Vehicles

Background

Sky

Aircraft

Tarmac

Grass

Sea

Car

Motorbike

Knowledge acquisition

N. Maillot 46/54

Results

Knowledge acquisitionHue Brightn

essGeometry Position Pattern

Aircraft Polygonal Center

Car Polygonal

Center

Motorbike Polygonal

Center

Sky Blue Grey

Dark Light

Top Smooth

Tarmac Grey Black

Bottom Uniform

Grass Green Bottom Uniform

N. Maillot 47/54

Results

• Use of the Caltech image database• Training Set : 850 images (aircraft, car, motorbike)• Test Set : 2000 images (contains 300 images of each class and 800 background images)

Background images

Images containing objects of interest

N. Maillot 48/54

Results: Caltech Database on 3 object classes

• Precision/Recall curve

Precision=n/ARecall=n/N

n: number of relevant retrieved images

A: number of retrieved images

N: number of relevant images

N. Maillot 49/54


Talk Overview

N. Maillot 50/54

Conclusion

• Approach: Use of ontological engineering combined with machine learning techniques

• Three phases:

1. Knowledge acquisition

2. Visual concept learning

3. Object Categorization

• Applications:

• Semantic image indexing and retrieval

• Knowledge acquisition in the domain of palynology

N. Maillot 51/54

• Contributions:• An extensible and reusable visual concept ontology

[maillot04]144 visual concepts (color, texture and spatial concepts)

• Original combination of knowledge and learning techniques for explicit domain knowledge elicitation and automatic visual concept detector learning [maillot04] In particular, no inference rules to define for mapping

• A weakly-supervised annotation approach [maillot05]enables easy image sample annotation

• An object categorization algorithm [maillot05]reproduces the way expert reasonindependent of the application domain

Conclusion

N. Maillot 52/54

Conclusion

• Strengths and Weaknesses:+ Elicitation of domain knowledge+ Reduction of the knowledge acquisition bottleneck

+ Reduction of the semantic gap

- Spatial reasoning missing- Image processing algorithms not adaptive- Geometric models not used during categorization

N. Maillot 53/54

Future Works

• Short-term:• Integration in a cognitive vision platform [Hudelot

05]• data management• top-down and bottom-up mechanisms• spatial reasoning

• Learning for adaptive image segmentation [Martin et al. 06]

• Long-term:• Extension to video content (e.g. temporal concepts)• Dynamic knowledge bases (no closed-world assumption)

• Use of 3-D models for categorization

N. Maillot 54/54

Thank you for your attention

N. Maillot

Publications

[1] Ontology Based Complex Object Recognition N. Maillot, M. ThonnatImage and Vision Computing JournalUnder Minor Revision

[2] Towards Ontology Based Cognitive Vision (Long Version) N. Maillot, M. Thonnat, A. BoucherMachine Vision and Applications Journal (MVA) Springer-Verlag Heidelberg, December 2004, 16(1), pp 33--40

[3] A Weakly Supervised Approach for Semantic Image Indexing and Retrieval N. Maillot, M. ThonnatInternational Conference on Image and Video Retrieval (CIVR 2005) Singapore, 20-22 July 2005

[4] Ontology Based Object Learning and Recognition : Application to Image Retrieval N. Maillot, M. Thonnat, C.Hudelot16th IEEE International Conference on Tools for Artificial Intelligence (ICTAI 2004)Boca Raton, Florida, 15-17 November 2004

[5] Towards Ontology Based Cognitive Vision N. Maillot, M. Thonnat, A. BoucherThird International Conference on Computer Vision Systems (ICVS 2003)Graz, Austria, April 2003, LNCS 2626, pp.44-53, Springer-Verlag Berlin Heidelberg 2003

N. Maillot 56/54

Proposed Approach

Data Management Knowledge Base

of Visual Concepts and Data

Data Management Engine

Interpretation Knowledge Base

of Application Domainand Visual Concepts

Interpretation Engine

Program SupervisionLibrary of

vision programs

Knowledge Base of Program Utilization

Program Supervision Engine

CurrentImage

Interpretation

ObjectHypotheses

Image Processing

Request

Numerical data

Image description

Visual Concept

Ontology

Cognitive vision platform [Hudelot 05]

ontology based object learning and recognition

Technology

object learning

knowledge acquisition

object categorization

object categorization

object identification

main object

object recognition summary

knowledgebased systems