dog i : an annotation system for images of dog breeds antonis dimas pyrros koletsis euripides...

DOGI: an Annotation System for Images of Dog Breeds

Antonis Dimas

Pyrros Koletsis

Euripides Petrakis

Intelligent Systems Laboratory

Technical University of Crete (TUC)

Chania, Crete, Greece

Image Annotation

The task of assigning a name or description to an unknown image

Manual: good quality, but slow, subjective Automatic: classification problem, relies on

associating image analysis features with high level concepts Difficult to handle all image types Semantic gap: map features to classes

04/19/23 ICIAR 2012, Aveiro, Portugal 2

DOGI: http://www.intelligence.tuc.gr/prototypes.php

An automatic image annotation system for images of dog breeds 40 classes (dog breeds) Descriptions: information in an ontology Class names, properties, features, textual

descriptions (from WordNet, Wikipedia) Annotations in MPEG7

04/19/23 ICIAR 2012, Aveiro, Portugal 3

DOGI: System

04/19/23 ICIAR 2012, Aveiro, Portugal 4

Graphical User Interface (GUI)

Feature Extraction

Ontology Mapping

Image Annotation

DOGI

Ontology

Load ImageSelect ROIAnnotation Method

MPEG7 features Color + Texture featuresImages: 12-dim vectors

40 classes9 instances/classClass hierarchy Class properties

Image RetrievalSelect Annotation MathodStore Annotation in Exif header

Select ROI

04/19/23 ICIAR 2012, Aveiro, Portugal 5

Image Content Analysis

Images of dog breeds are mainly characterized by the spatial distribution of color intensities

A 12-dimension feature vector of Color, Texture, Hybrid feature from LIRE Library

Features are normalized in [0,1] Not all features are equally important

04/19/23 ICIAR 2012, Aveiro, Portugal 6

Ontology

40 classes of dog breed organized in IS_A hierarchy E.g., Dog Working Group Saint Bernard

Three separate hierarchies for text, features and visual descriptions

9 instances per class: raw images + a 12-dim feature vector for each image in class

04/19/23 ICIAR 2012, Aveiro, Portugal 7

DOGI Ontology

04/19/23 ICIAR 2012, Aveiro, Portugal 8

Image Annotation

The unknown image Q is compared with each one of the 360 images in the ontology

D(Q,I) = Σi widi(Q,I)

Results are ranked by similarity with Q Weights wi are computed by decision trees

Training set of 3,474 image pairs

04/19/23 ICIAR 2012, Aveiro, Portugal 9

i feature of nodes

1

i

)(1maxdepth

)uredepth(feat - 1 maxdepth

nodesall

j j

inodedepth

w

Annotation Method

Best Match: Select class of most similar instance

Max Occurrence: Select class with more instances in the first 20 answers

Average Retrieval Rank: Select class with instances ranked higher in the first 20 answers

Max Similarity: Select class whose instancing sum-up to max similar score

04/19/23 ICIAR 2012, Aveiro, Portugal 10

Example Image

04/19/23 ICIAR 2012, Aveiro, Portugal 11

Annotation Result

04/19/23 ICIAR 2012, Aveiro, Portugal 12

EXIF Metadata

Descriptive information embedded inside an image

The metadata captured by your camera is called EXIF data ..

DOGI stores annotation info with the pictures in the EXIF

Can be useful for image archiving and later retrievals

04/19/23 ICIAR 2012, Aveiro, Portugal 13

Annotation in MPEG7

04/19/23 ICIAR 2012, Aveiro, Portugal 14

Evaluation

Average annotation accuracy over 40 queries

04/19/23 ICIAR 2012, Aveiro, Portugal 15

Annotation result

Max Similarity

AVR Max Occurrence

Best Match

Ranked 1st 72.5% 62.5% 65% 50%

Ranked 2nd 17.5% 22.5% 15% 10%

Ranked 3rd 5% 10% 10% 10%

Overall 95% 92,5% 90% 90%

Conclusions-Future Work

DOGΙ : An automatic annotation system for dog breeds with good performance

Useful as a tool for many application Annotation accuracy improves for less

categories Experimenting with more and animal species

images categories More elaborate image classification methods

04/19/23 ICIAR 2012, Aveiro, Portugal 16

THANK YOU !!

04/19/23 ICIAR 2012, Aveiro, Portugal 17