mathieu delalandre 1 , ernest valveny 1 , tony pridmore 2 1 cvc, barcelona, spain

Graphics Programming of Constraint for Object Positioning:

Application to Performance Evaluation of Symbol Recognition & Spotting Systems

Mathieu Delalandre1, Ernest Valveny1, Tony Pridmore2

1 CVC, Barcelona, Spain2 SCSIT, Nottingham, UK

CVC SeminarCVC, Barcelona, Spain

Friday 27th of June 2008

Introduction

symbolbackgroundtext

Recognition

Spotting

r1 r2 r3

sofa

skin

tubdoordo

ordocument database

learning

database

Query By

Example

(QBE)

rank

labels

Symbol spotting: “a way to efficiently localize possible symbols and limit the computational complexity, without using full recognition methods” [Tombre2003] [Dosch2004] [Tabbone2004] [Zuwala2006] [Locteau2007] [Qureshi2007] [Rusinol2007]

Symbol recognition: ““a particular application of the general problem of pattern recognition, in which an unknown input pattern (i.e. input image) is classified as belonging to one of the relevant classes (i.e. predefined symbols) in the application domain” [Chhabra1998][Cordella1999] [Llados2002] [Tombre2005]

Electrical diagram

Mechanical drawing

Utility map

scanned

CAD file Web image

Introduction

Characterisation

GroundtruthGroundtruthGroundtruth

Groundtruthing

ResultsResultsResults

Performance evaluation

System

Performance evaluation: Information Retrieval [Salton1992], Computer Vision [Thacker2005], CBIR [Muller2001], DIA [Haralick2000]

IntroductionReal approach

Document

Document

Document

Groundtruth

Groundtruth

Groundtruth

Groundtruthing

- - weak ++ good real

approach

synthetic

approach

Groundtruthing methods for symbol recognition/spotting:

GTground-truthing

validation

groundtruth

drawings and alerts

groundtrutheddrawings

validation and

alertsevaluationtest images

recognition

results

Dosh and al 2006

10

43

2

5

4

01

5

32connectedparallel and

overlapped

Yan and al 2004

IntroductionSynthetic approach

Document

Document

Document

Groundtruth

Groundtruth

GroundtruthGroundtruthingSetting

- - weak ++ good

real approach

synthetic

approach

Groundtruthing methods for symbol recognition/spotting:

Aksoy 2000

binary noise

vectorial noise

Valveny and al 2007Zhai and al 2003

symbolbackground

Graphical documents are composed of two layers

To use a same background layer with different symbol layers

To associate symbols to positioning constraints and to select them at random

Key appraoch

c2

c1

Problematic, how to build whole synthetic documents ?

First constraint, how to deal with the knowledge level ?

-

++

-

2-connected

1-connected

1-connected1-connected

2-connected

Production rules0-n

0-n

11

O-n

‘Creasy’ but well formed drawing

Introduction

Second constraint, how to make usable the system ?

Symbol recognition people

Database

chemical

symbol

network

symbols

Plan1. Positioning constraint2. Building system3. Graphics User Interface4. Experiments5. Conclusions and perspectives

1. Positioning constraint2. Building system3. Graphics User Interface4. Experiments5. Conclusions and perspectives

Positioning constraint

Model dependent constraint

a modelconstraintsto include

to connect to

border

Zone

ddL x

cos2

ddL x

cos2

d

dL y

2cos2

2cos2 d

dL y

2

x

y

dd

a tan

x

y

dd

a tan2

(2)(3)

(4)Ld

(1)(1)

(2)

(3)

(4)

How to get the length l from ρ ?

dx

dy

Ll (2)(3)

(4) d

(1)

dx

(4)

(3)

ϴ

(1)(2)

1 (4)dy

1

x

yd

dad tantan(1)

(2)

(3)

(4)

y

xd

dad 2

tantan2

y

xd

dad2

tantan2

x

yd

dad tantan

How to get the direction d from θ ?

How to define the constraint ?Our approach, model independent

control point anchor point selection

(3) positioning on background

(1) computation of control point

(2) sloping+1

-1

-1+1 ρ = 1 ; = 0.125

ρ

Unit polar coordinate to define a control point inside a bounding box


Positioning constraint

How to select the anchor point ?1. Just a fixed value (x,y)

x0

y0

y0+dy

x0+dxx0+rxdx

y0+rydy

0 rx,y 1

3. Taken at random inside a zone

0

10,5

r

2. Taken at random belong a line

x,y

x’,y’

dx

dy

l

lxx cos' lyy sin'

(a) (b)

01

0,5l = r L

Lb

e

rpl

point

0

01

The polygon includes the point if | 01 + 12 + …… 60 | = 2π

3 4

5

62

112

+

-

d1

d2

d2

021 dd

2,0,

,2,0

2π -

d1 and d2 are the two lines’ directions

4. Other shapes (arc, curve, ellipsis etc.)


Building system

model selection

full constraint

overflow

constraint selection

symbol factory & positioning

overlap

continue

constraint stack empty

build

ing

failu

re

symbol and constraint

continue

object and constraint

continue

continue

no yes

building end

new loop

(1) symbol and constraint selection

(2) symbol factory and positioning

(3) constraint checking

(5) Stopping criterion

(4) building checking


S1

S2

S3

S4

C1

c2

c1C2

C3

C4

wc1 = 0.33

wc2 = 0.33

wc3 = 1.00

ws1 = 0.33

ws2 = 0.33

ws3 = 0.66

ws4 = 0.33

ws5 = 0.33

ws6 = 1.00

ps1 = 0.11

ps2 = 0.11

ps3 = 0.22

ps4 = 0.11

ps5 = 0.11

ps6 = 0.33

sc nw 1

acn

icis ww

1

c

ss nwp

s1

s2

s3

s4

s5

s6

c1

c2

c3

(c)onstraint and (s)ymbol

symbol based selectionconstraint based selectionci s1

s2s3ci-1

ci-2…


model selection

full constraint

overflow



overlap

continue


build

ing

failu

re


continue


continue

continue

no yes

building end

new loop

(1) symbol and constraint selection

Building system

x,y

x’,y’

dx

dy

l lxx cos' lyy sin'

arc is directed to because is lower than γ

02Π

γ

trigonometric

d1

d2

d2

021 dd

2,0,

,2,0

2π -

d1 and d2 are the two lines’ directions

Building system

begin

end

width

begin

end

center

width width

centerradius

model selection

full constraint

overflow



overlap

continue


build

ing

failu

re


continue


continue

continue

no yes

building end

new loop

(2) symbol factory and positioning


control point anchor point selection

(4) positioning on background

(2) computation of control point

(1)rotation scaling, morphing

(3) sloping

model selection

full constraint

overflow



overlap

continue


build

ing

failu

re


continue


continue

continue

no yes

building end

new loop

(3) constraint checking

Full constraintEach constraint have a number of symbol to push

nNSP ,1

p1 p2

p3p4

symbol overflows P if P doesn’t include p1 or p2 or p3 or p4

PP

(a) (b)

01

0,5l = r L

Lb

er

pl point

0

01

The polygon includes the point if | 01 + 12 + …… 60 | = 2π

3 4

5

62

112

+

-

Building system1. Positioning constraint2. Building system3. Graphics User Interface4. Experiments5. Conclusions and perspectives

L covers L if e and b are overlapped

b e

L

left

bottom

right

up

symbol overflows L if right or up or left

or bottom is not covered by L

L overlap p if dx1 dx2 < 0 or dy1 dy2 < 0

dx1 dx2

dy1 dy2p

L

L overlap p if dx1 dx2 < 0 or dy1 dy2 < 0

dx1 dx2

dy1 dy2p

L

L overlaps L if e or b are overlapped

b e

Boxes are overlapped if L1 overlaps L2 or L2 overlaps L1

and L3 overlaps L4 or L4 overlaps L3

L1

L2L3

L4

Layer 2with overlap

Layer 1without overlap

background

model selection

full constraint

overflow



overlap

continue


build

ing

failu

re


continue


continue

continue

no yes

building end

new loop

(4) building checking

Building system1. Positioning constraint2. Building system3. Graphics User Interface4. Experiments5. Conclusions and perspectives

ContributionsBuilding Manager

ContinueWhen building failures (overflow, full, overlap) become upper to number of wished symbol we stop the process

k

iiNSP

1

model selection

full constraint

overflow



overlap

continue


build

ing

failu

re


continue


continue

continue

no yes

building end

new loop(5) Stopping criterion


Graphics User Interface

Constraint Setting

Model Selector

Constraint Selector

Model Viewer

Building Bar

Constraint Locator

Building Viewer

1. Background broswing

2. Model loading

3. Model viewing

4. Constraint edit

5. Constraint linking

6. Building viewing

7. Constraint import/eport

8. Resolution setting

…


SymbolModels

BuildingEngine

Run

View

Edit

Model loading, selection and viewing

loading

select‘scroll & click’

automatic zooming

attach

focus‘click and well’

shift‘drag’

Background attachment and browsing

Graphics User Interface1. Positioning constraint2. Building system3. Graphics User Interface4. Experiments5. Conclusions and perspectives

‘click’

‘click, move, click’or’

‘pressed, drag, release’

‘click(s), double click’

Locate the constraints

select‘click’

link model

add‘right double click’

delete‘left double click’

Link the constraints


Control

Slope


Experiments1. Positioning constraint2. Building system3. Graphics User Interface4. Experiments5. Conclusions and perspectives

dataset

images

symbols

models

bags #16 1600 15046

25-150

floorplans

#10 1000

26830 16

#26 2600

41876


dataset

images

symbols

models

bags #16 1600 15046

25-150

floorplans

#10 1000

26830 16

#26 2600

41876

without rotation,without scaling

with rotation, without scaling

without rotation, with scaling

with rotation, with scaling


dataset

images

symbols

models

bags #16 1600 15046

25-150

floorplans

#10 1000

26830 16

#26 2600

41876

Conclusions and Perspectives

• ConclusionsFirst valid system to build whole graphic synthetic

documents+ Built documents kept meaning and seem realistic+ No constraint on database sizes (thousands …..)+ Contents of documents can be controlled + Can be use on several backgrounds with “reasonable” work costs using the GUI (1-2

hour per background)



Conclusions and Perspectives• In progress, perspectives

Spotting/RecognitionSystem

Groundtruth Mapping

Region Of

Interest

Characterization

sofa

skin

tubdoor

door

Labels

r1 r2 r3

Ranks

QBE

truth results

Learning

1. Evaluation objectives2. Campaign organization3. Background types4. Model & domain

scalability5. Sizes of image and

databases6. Degradation models7. System training

(learning data, QBE)8. Groundtruth data & file9. Mapping algorithm10. Recognition & ranking11. System taxonomy12. System profiling

Forum document

LITIS

Rouen

CVC Barcelona

SCSIT Nottingham

LI Tours

LORIANancy

Alicia Fornes, Dimosthenis Karatzas, Hervé Locteau, Jean-Pierre Salmon, Jean-Yves Ramel, Marçal Rusinol, Philippe Dosch, Rashid Qureshi and Tony Pridmore

Workgroup

ThanksBibliography1. M. Delalandre, E. Valveny and J. Lladós Performance Evaluation of Symbol

Recognition and Spotting Systems: An Overview. Accepted paper to the Eighth IAPR Workshop on Document Analysis Systems (DAS), 2008.

2. M. Delalandre, T. Pridmore, E. Valveny, H. Locteau and E. Trupin. Building Synthetic Graphical Documents for Performance Evaluation. Revised Selected Papers of Workshop on Graphics Recognition (GREC), Lecture Notes in Computer Science (LNCS), vol 5046, pp 288-298, 2008.

mathieu delalandre 1 , ernest valveny 1 , tony pridmore 2 1 cvc, barcelona, spain

Documents

graphics user interface4

positioning constraint2

object positioning

perspectives experiments1

constraint edit5

constraint linking6

constraint importeport8

symbol recognitionspotting