script independent keyword spotting using moment featuresword spotting (gabor) previous work...
Post on 24-Mar-2020
6 Views
Preview:
TRANSCRIPT
Jan 11, 2008
Script Independent Keyword Spotting Using Moment Features
Venu Govindaraju
Venu@cubs.buffalo.edu
Jan 11, 2008
Keyword Spotting for Multi-script Documents
Image – Based
Query Image
Provided OR
Rendered
OCR - Based
Match in Feature Space
Jan 11, 2008
Multi-script Documents
Challenge: Script Invariant Word Image Representation
Jan 11, 2008
Devanagari OCR(Block Adjacency Graph)
Branching
Merging
Sample stroke Graph representation
BAG of conjunct character
BAG of alphabets
……
Segmentation Hypothesis
Jan 11, 2008
Recognition Methodology
Words
….
• Language model to choose path - results:– Script writing grammar rules eliminate two choices:
– Phonetic n-gram constraints remove another choice:
Jan 11, 2008
Word Spotting (GSC)Previous Work
Matching in feature space [Srihari, et al, SPIE 2004]
Matching GSC features of two word images.
Corpus: 9312 word images (3104 for queries and 6208 for tests) from 776 individuals
PerformanceReport GSC outperforms DTW
1024-bit GSC feature
Jan 11, 2008
Word Spotting (DTW)Previous Work
Word image
Upper/ lower
profile features
Observdensity ),Pr( fvwrd
Word recogn
Prob ∑=
wrdfvwrd
fvwrdfvwrd
),Pr(),Pr(
)|Pr(
Word recognition probability• Sequential Profile and DTW
[Rath et al, CVPR 2003]
CorpusWashington’s manuscripts
PerformanceAverage precision: 67.92%QueryImage and Text
Jan 11, 2008
Word Spotting (Gabor)Previous Work
Template Free Word Spotting in low-quality manuscripts [Huaigu, et al, ICAPR 2007]
Matching Gabor features of two word images. Corpus:
12 medical forms containing 5295 character images.
PerformanceReport probabilistic similarity performs better than Euclidean
similarity and WMR.
Feature Extraction
V1 V2 V3 V4
Vw = [V1T V2T V3T V4T]T
))|(ln(1),(1∑=
=
−=ni
iiP vcP
nVwwC i
Probabilistic Similarity
Jan 11, 2008
Issues• Deal with
– Complex characters (Devanagari)– Scale and translation– Multiple scripts
• Structural Features (GSC)– Script specific therefore ineffective
(Srihari et al)
– Profile features applicable only on long components (Manmatha et al)
Ascenders
Descenders
Core
Shirorekha
Base line
Jan 11, 2008
Moment Features Geometric Moments
Center of Gravity
Central Moments
Variance
Jan 11, 2008
Moment Features• Pre-processing of document images
• Moments (up to 7th order) extracted from normalized word images
• Invariant to scale and translations
Feature vector = [
230.045291568.635182-10.828037
-3200.1645835531.698681
-23057.359438202901.278885
-42801047.364910-28009097.448910755255816.89022
….….
]Feature vector consists of 30 moment values Construct for each word image and store in the index
Jan 11, 2008
4 6 8 10 12 14 160.7
0.72
0.74
0.76
0.78
0.8
0.82
0.84
0.86
Moment Order
Avg
Prec
.
Avg. Precision Vs Moment Order
Noise Sensitivity-High Order MomentsHindi Dataset
Average Precision vs higher order moments, Apply relevance feedback to re-rank word images
Jan 11, 2008
Corpus
Hindi : 763 machine print word images extracted from Million Book Project documents.
English : 707 handwritten word images extracted from IAM database and George Washington’s historical manuscripts.
Sanskrit : 693 machine print word images extracted from 5 documents downloaded from the URL : http://sanskrit.gde.to/
Jan 11, 2008
Corpus
• The test corpus consists of 5780 word images extracted from Million Book Project Documents.
Jan 11, 2008
Keyword Spotting• ja.ngal [ -3060.48 , 710.86 , 480388.32 , -43156.29 , ]
• kabUtar [ 31000.55 , 2774.74 , 496660.19 , 7229.75 , -]
• Vachan [ 8208.35 , -2379.97 , 146283.25 , 4141.59 , ]
Jan 11, 2008
Cosine Similarity
English Hindi
Pair A B CA 1 0.9867 0.9932B 0.9867 1 0.9467C 0.9932 0.9467 1
Pair D E FD 1 0.9662 0.9312E 0.9662 1 0.9187F 0.9312 0.9984 1
D
E scale
F Linear
A
B Scale
C Linear
Jan 11, 2008
Query OCR GSC Gradient GaborBaba 0.62 0.8 0.41 0.41
Bandar 1.0 0.092 1.0 1.0Bhakt 0.75 0.52 0.41 0.41Jungal 0.65 0.74 0.1 0.1Machali 0.79 0.93 0.89 0.89
Mata 0.63 0.35 0.38 1.0Naksha 1.0 0.77 1.0 0.41
Raat 0.70 0.72 0.41 0.38Ramkumar 1.0 0.89 0.22 0.24
Vachan 0.86 0.67 0.24 0.22
Mean for 10 0.80 0.65 0.50 0.50Mean for 20 0.67 0.60 0.29 0.29
Mean Average Precision
Jan 11, 2008
Relevance Feedback
Relevance
query
results
Query Feature
Word Features
Ranking
Feedback
INDEXINGDocImages
Jan 11, 2008
Relevance Feedback on Vector Space
R1 = [ 12333.37 , -12148.82 ]
Qnew = [ 1288.43 , -8450.10 ]
Q1 = [ -3060.48 , 710.86 ]M21
NR1 = [ 31000.55 , 2774.75 ]
NR2 = [ 8208.35 , -2379.98 ]
M12
Jan 11, 2008
Relevance Feedback
Q1 = [ -3060.48 , 710.86 ]
R1 = [ 12333.37 , -12148.82 ]
NR1 = [ 31000.55 , 2774.75 ]
NR2 = [ 8208.35 , -2379.98 ]
Qnew = Q1 + 0.75 * (R1) – (0.25/2) * (NR1 + NR2)
Qnew = [ 1288.43 , -8450.10 ]
Cosine Similarity (Q1 , R1 ) = - 0.8527
Cosine Similarity (Qnew , R1 ) = 0.8011
Jan 11, 2008
Average PrecisionScript W/O Relevance
Feedback %Relevance Feedback %
English 66.30 69.20Hindi 71.18 74.34Sanskrit 87.88 92.33
Script GSC50% Recall
Moments50% Recall
English 60.0 71.6Sanskrit 90.0 94.3
Script Gabor MomentsEnglish 56.15 66.30Hindi 67.25 71.18Sanskrit 79.10 87.88
Jan 11, 2008
Average Precision
1 2 3 4 5 6 7 80.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Query
Avg
Prec
isio
nAverage Precision Curve for few Queries English
0 10 20 30 40 50 600.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Queries
Aver
age
Prec
ision
Avg. Precision for few Queries
0 10 20 30 40 50 60 70 80 90 1000.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Queries
Aver
age
Prec
ision
Avg Precision for few Queries
SanskritHindi
8 queries
75 queries
100 queries
Jan 11, 2008
Summary
• Keyword Spotting Methods– OCR driven– Image based: Moments, GSC, Gabor
• Multi-script Documents– Moments– Relevance Feedback
• Future Work
top related