building synthetic graphical documents for performance evaluation of symbol recognition m....

Building Synthetic Graphical Documents for Performance Evaluation of Symbol Recognition

M. Delalandre1, E. Valveny1 and T. Pridmore2 1 CVC, Barcelona, Spain2 SCSIT, Nottingham, UK

SymbolRec MeetingCVC, Barcelona, Spain

Thursday 13th December 2007

Plan

• Introduction• Building Engine• Constraint Editor• Use Cases• Conclusions

Introduction (1/4)Graphics Analysis Systems and Performance Evaluation

AnalysisDocument

Document

Document

Characterisation

Groundtruth

Groundtruth

Groundtruth

Groundtruthing

Results

Results

Results

Performance evaluation

Evaluation

Performance evaluation

1. Analysis and Evaluation2. Groundtruthing3. Problematic & Approach4. Our System

Symbol recognition systems

results label + localization

SymbolRecognition

spotting, retrieval, understanding, …

Introduction (2/4)Groundtruthing

Document

Document

Document

Groundtruth

Groundtruth

Groundtruth

Groundtruthing

Based on real-life documents

Advantages : based on real life documentsDrawbacks : copyright, cost, length, error, content control

Akoy’00

binary noise

vectorial noise

Valveny’03

Based on synthetic documents

Document

Document

Document

Groundtruth

Groundtruth

GroundtruthGroundtruthingSetting

Advantages : easy to produce, few constraintDrawbacks : not real

Past works

1. Analysis and Evaluation2. Groundtruthing3. Problematic & Approach4. Our System

Introduction (3/4)Problematic & Approach

1. Analysis and Evaluation2. Groundtruthing3. Problematic & Approach4. Our System

symbol

background

Graphical documents are composed of two layers

To use a same background layer with different symbol layers

To associate symbols to positioning constraints and to select them at random

Our approach

c2

c1

ProblematicHow to build whole documents ?

How to deal with the real in the synthetic ?

How to make an usable approach ?

-

++

-

2-connected

1-connected

1-connected

1-connected

2-connected

Production rules

0-n

0-n

1

1

O-n

‘Creasy’ but well formed drawing

Introduction (4/4)Our System

GTGT

GTGT

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

Four components• The Engine

• Symbol Model & Factory• Symbol Positioning• Building Manager

• Constraint Editor

Plan

• Introduction• Building Engine• Constraint Editor• Use Cases• Conclusions

Building EngineSymbol Models & Factory

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

(x,y)begin

end

width

begin

end

center

width width

center

radius

Symbol models

One file, one model

Vector graphics format(point, line, arc, circle)

Symbol factory

x,y

x’,y’

dx

dy

l lxx cos'

lyy sin'

arc is directed to because is lower than γ

02Π

γ

trigonometric

d1

d2

d2

021 dd

2,0,

,2,0

2π -

d1 and d2 are the two lines’ directions

Building EngineConstraints & Symbol Positioning

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

cos2xdl

cos2 xdl

2cos2

ydl

2cos2

ydl

2

x

y

d

da tan

x

y

d

da tan

2

dx

dy

(2)(3)

(4)l

(1)

(1)

(2)

(3)

(4)

How to select the control point ?

Define for each constraint by polar unit coordinate (ρ,) computed from bounding box

0

10,5

r

How to select the anchor point ?

(a) (b)

0

1

0,5l = r L

Lb

e

rpl

point

0

01

The polygon includes the point if | 01 + 12 + …… 60 | = 2π

3 4

5

62

112

+

-x0

y0

y0+dy

x0+dxx0+rxdx

y0+rydy

0 rx,y 1

Just a fixed value (x,y)

Taken at random belong

a line

Taken at random inside a zone

control point anchor point selection

(4) positioning on background

Constraint, main principle

(2) computation of control point

(1) rotation (3) sloping

Building EngineBuilding Manager (1/2)

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

L overlap p if dx1 dx2 < 0 or dy1 dy2 < 0

dx1 dx2

dy1 dy2p

L

L overlaps L if e or b are overlapped

b e

Boxes are overlapped if L1 overlaps L2 or L2 overlaps L1

and L3 overlaps L4 or L4 overlaps L3

L1

L2L3

L4

Symbol overlapping

L covers L if e and b are overlapped

b e

L

left

bottom

right

up

symbol overflows L if right or up or left

or bottom is not covered by L

Line overflow

p1 p2

p3p4

symbol overflows P if P doesn’t include p1 or p2 or p3 or p4

P

P

Zone overflow

sc nw

1

acn

icis ww

1

c

ss n

wp

wc1 = 0.33

wc2 = 0.33

wc3 = 1.00

ws1 = 0.33

ws2 = 0.33

ws3 = 0.66

ws4 = 0.33

ws5 = 0.33

ws6 = 1.00

ps1 = 0.11

ps2 = 0.11

ps3 = 0.22

ps4 = 0.11

ps5 = 0.11

ps6 = 0.33

s1

s2

s3

s4

s5

s6

c1

c2

c3

(c)onstraint and (s)ymbol

Symbol selection Stopping criterionWhen building failures (overlapping, overflow, capacity) become upper to number of wished symbolwe stop the process

Plan

• Introduction• Building Engine• Constraint Editor• Use Cases• Conclusions

Constraint Editor Introduction

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

Constraint Setting

Model Selector

Constraint Selector

Model Viewer

Building Bar

Constraint Locator

Building Viewer

SymbolModels

BuildingEngine

Run

View

Edit

Constraint Editor Background and Models

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

Model loading, selection and viewing

loading

select‘scroll & click’

automatic zooming

attach

focus‘click and well’

shift‘drag’

Background attachment and browsing

Constraint Editor Locate and Link the Constraints

‘click’

‘click, move, click’or’

‘pressed, drag, release’

‘click(s), double click’

Locate the constraints

select‘click’

link model

add‘right double click’

delete‘left double click’

Link the constraints

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

Constraint Editor Building View

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

view‘CRTL+V’

view

focus‘click and well’

shift‘drag’

keeping view

Viewing and browsing Some views

Constraint Editor Wrap, Scale and Rotate

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

Constraints

Wrap

1.0

1.2

1.5

Scale Rotate

Constraint Editor Control and Slope

SymbolModels

Building Manager

SymbolFactory

SymbolPositioningPositioning

ConstraintsControl

Slope

Plan

• Introduction• Building Engine• Constraint Editor• Use Cases• Conclusions

Use CasesWork Group

Marçal Rusinol (CVC), Ernest Valveny (CVC), Mathieu Delalandre (CVC), Jean-Yves Ramel (LI), Rashid Qureshi (LI), Philippe Dosch (LORIA), Tony Pridmore (SCSIT) and Hervé Locteau (LITS).

http://docs.google.com/Doc?id=dgjczxf6_4ccc8kh LITIS

Rouen

CVC

Barcelona

SCSIT

Nottingham

LI

Tours

LORIA

Nancy

Use CasesArchitectural drawings

Dataset: 1000 images, 18 000 symbols, 16 models Constraints: connectivity and heterogeneous backgrounds

•10 backgrounds•filled, empty, textured

Use CasesBags of symbol

Dataset: 1600 images, 15 000 symbols, 150 modelsConstraints: scalability, rotation and scaling

• rotation & scaling• 25,50, 100 and 150 models

16 bases 100 images (1024)2 / base 10 symbols / image

Conclusions and Perspectives

• ConclusionsFirst valid system to build whole graphic synthetic documents

• Built documents kept meaning and seem realistic• No constraint on database sizes (Thousands …..)• Contents of documents can be controlled • Can be use on several backgrounds with “reasonable” work costs using the GUI (1/4 hour per background)

• Perspectives• Databases under packaging for GREC contest 2007• To build different document types (diagram, maps)• To propose a characterization algorithm working with the produced

groundtruth