Indexing Text with Approximate q-grams
Adriano Galati & Marjolijn Elsinga
Overview
• Approximate string matching- Neighborhood generation- Reduction to Exact Searching- Intermediate Partitioning
• Indexing text using q-grams• Filtration condition• Finding approximate q-grams
- Trie data structure- Non-deterministic automaton (NFA)
• Parameters
Approximate string matching
Text
Pattern
Goal: Retrieve all occurrences of P in T whose edit distance is at most k
Edit distance: ),( BAed
nT ..1
mP ..1
Solutions
All kinds of solutions, most investigated area in computer science
In on-line versions of the problem the pattern can be preprocessed, the text cannot
Classical solution: using dynamic programming and a matrix is O(mn)
Fill matrix where ìs the minimum edit distance between P and a suffix of T
Initialize the borders with
Fill internal cells with
Classical solution
nmC ..0,..0 jiC ,
0 and ,00, ji CiC
),,min(1
if
1,1,1,1
1,1
jijiji
jiji
CCC
TPC
Solution (2)
If text is large, on-line algorithms are not practical and preprocessing becomes necessary
Focus: Sequence retrieving indexes, with no restrictions on the patterns and the occurrences
Approaches:• Neighborhood Generation• Reduction to Exact Searching• Intermediate Partitioning
Neighborhood Generation
Set of strings matching a pattern with k errors is finite ( )
Therefore it can be enumerated
Each string can be searched using a data structure
This structure is designed for exact matching
)(PU k
)(PU k
Neighborhood Generation (2)
+ O(n) space and construction time- Not optimized for secondary memory- Inefficient in space requirements
Is promising for searching short patterns only
Reduction to Exact Searching
Indexes based on filters
Filter checks for simpler condition than the matching condition, discarding large parts of the text
Main principle: if two strings A and B match with k errors and k+s non-overlapping samples are extracted from A, then at least s of these must appear without errors in B
Reduction to Exact Searching (2)
+ can be built in linear time and need O(n) space
+ with some method it is possible to make an index that takes less space then the text itself
- Are based on suffix trees or on indexing all the q-grams
Intermediate Partitioning
Reduces the search to approximate search instead of exact search
Main principle: if two strings A and B match with at most k errors and j disjoint substrings are taken from A, then at least one of these appears in B with
Split the pattern in j pieces, search each piece in the index allowing errors, extend the approximate matches to complete occurrences
jk /
jk /
Question (Ingmar)
I think the main principle is incorrect, because if AAABBBBBB
BBBBBBBBB
These match with k=3 errors. If we take the disjoint substrings AAA BBB BBB so j=3. Now they say that one of these will appear in the other with errors. However AAA match with 3 errors, BBB with 0 and BBB with 0
13/3
Answer
The pattern is split in j pieces, each piece is searched in the index allowing errors
AAA BBB BBBBBB BBB BBB
We match BBB with ABB and not with AAA and AAB, because it is not possible to match them with more then errors, with k=3 and j=3, unless we change the parameters
jk /
jk /
Intermediate Partitioning (2)
+ optimizing point between neighborhood generating (worse with longer pieces) and reduction to exact searching (worse with shorter pieces)
Has been used on the patterns but not yet on the text itself
Indexing text using q-grams
Steps:• Filtering text• Finding approximate q-grams
Advantages:• Takes little space• Has an alternative tradeoff• User can decide what is important: saving space
or better performance
Filtration condition
Based on locating approximate matches of pattern q-grams in text
Leads to a filtration tolerating higher error levels compared to exact q-gram matching
Condition for an approximate match
Two strings A and B
Now: at least one string Ai appears in B with at most errors
Only the q-grams for which this hold, will be used for searching
kBAed ),(
jj AxxAxAA 12211 ...
jk /
Example: Condition
A: CCTC TCTC CCCT B: CCCC CTCT TCTC
We see: k=8We take: j=3
Now e=2, so at least one Ai appears in B with at most 2 errors
Question (Peter)
“Note that it is possible that , so we are not only ‘distributing’ errors across pieces, but also ‘removing’ some of them”
How does this work?
kjkj /
Answer
A1 A2 A3x1 x2
k=5
j=3
e=1
Q-grams vs. Q-samples
Q-grams overlapQ-samples do not overlap
String: ABCDEFQ-grams: {ABC, BCD, CDE, DEF}Q-samples: {ABC, DEF}
In a q-gram index all the text q-grams are stored in increasing order
In a q-sample index only some text q-grams are stored
Constructing q-samples
We need to extract j pieces from each potential pattern occurrence in the text
So: a q-sample every h text-characters
We need to guarantee that j q-samples are inside any occurrence of P
Minimal length of P = m-k
j
qkmh
1
Question (Jacob)
Could you please explain how the restriction of h is built up?
Answer
j
qkmh
j
qkm
j
n
qkmn
qPn
q
1
1
1
1
1textsamples-q#
Next step
Best match distance (bed) is calculated for each test sequence of q-samples
This is the distance between the q-sample sequence and the involved text (h)
The text area h is only examined if its bed is at most k
Algorithm
Each q-sample sequence has its own counter M
M indicates the number of errors produced by the q-sample sequence and is initialized to
So: we start by assuming that each q-sample gives enough errors to disallow a match
)1( ejM
Error-environment
After calculating the M for each q-sample sequence, we obtain the e-environment of each q-sample sequence
This is the set of possible q-samples that appear inside the q-sample sequence with at most e errors
Finishing
Now all text areas have its own e-environments connected to it through the q-samples
They can be checked with dynamic programming
Finding approximate q-grams
Finding all the text q-samples that appear inside a given pattern block
Note: it is not necessary to generate all since we are interested only in the text q-samples (position)
( ) { 1.. / , ( , ) }qe r iI Q r n h bed d Q e
iQ
( )qe iU Q
Finding approximate q-grams (2)
Idea: to store all the different text q-samples in a trie data structure
We fill in a matrix such that is the sed between and a suffix of
is relevant for some In a trie traversal of the q-samples, the characters
of are obtained one by one
l1..iS 1..lQ
S ,q lC e l
S
0.. ,0..| |q QC
Question (Laurence)
Can you please show me the matrix is build on page section 3 in fig. 4? It is a bit unclear to me how the matrix is initialized and the different cells are being filled.
Answer
i jS Q,i jC if then 1, 1i jC else 1, 1, 1 , 11 min( , , )i j i j i jC C C
Answer
s
u
r
v
e
y
1
2
3
0
1
2
34
5
0 0
4
56
s
1
2
1
1
2
2
21
2
0 0
1
22
eg
1
0
1
1
1
0
12
3
0 0
2
34
ru
1
2
2
1
2
3
33
2
0 0
3
22
yr
Finding approximate q-grams (4)
When we reach the leaf nodes (depth q) we check in if there is a cell with value the corresponding text is reported
Complexity
e
(| | ) ( )O Q q O mq
Finding approximate q-grams (3)
Pruning:• All the value of a row to the next are nondecreasing• If all the values of a row are larger than at that
point we can abandon that branch of the triee
Finding approximate q-grams (5)
Alternative way:• To model the search with a non-deterministic
automaton (NFA)
Finding approximate q-grams (6)
Consider the NFA for errors
Every row denotes the number of errors seen
Every column represents matching a prefix of S
Horizontal arrows represent matching a character
All the others increment the number of errors
2e
Question (Bogdan)
I can imagine how the trie can be used together with the matrix in order to benefit from common prefixes of certain q-samples (by reusing the rows of the matrix which are already computed for the common prefix). However, I don't see how this can be done in the case of the NFA. If it can't be done, this would mean that the algorithm has to be run separately for each q-tuple, which probably makes the NFA approach much worse. Am I right to think that or is there a way to run the NFA in a "smarter" way so as to benefit from common prefixes?
Bogdam (answer)
Yes, you are right, the algorithm has to run for each q-tuple, but you have to consider the complexity of it, that is linear ( )O e
Parameters of the Problem
Smaller value the search of e-environment will be cheaper
Larger value gives more exact estimates of the actual number of error but with a higher cost to search the e-environment
As grows, longer test sequences with less errors per piece are used the cost to find the relevant q-samples decreases but the amount of text verification increases.
e
e
j
Parameters of the Problem (2)
1. Notice: the index of this approach only stores non-overlapping q-samples, its space requirement is small
2. Notice: the space consumption of index depends on the interval h
Parameters of the Problem (3)
Standard implementation q-gram index stores all the locations of all the q-grams of the text
The number of q-grams Storing a position takes space consumption is Ratio between this method and standard
approach
1n q
log n
logn n
/ log( / ) 1
logr
n h n hv
n n h
Question (Bogdan)
Could you please explain what the "columns" used in the 5th section are?
The table shows how the error level increases the number of processed columns of matrix or NFA
Question (Lee/Bram)
The article talks about disjoint non overlapping q-grams. At the end they say that will probably enhance the scheme that they allow overlapping q-grams. Any idea how our current algorithms have to be changed for that and what the advantages are?
http://www.cs.utexas.edu/users/mobios/MoBIoSPapers/2003-IndexingProteinSequences-TR-04-06.pdf
Question (Lee)
In the second paragraph of section 4 they say “In that particular case we can avoid the use of counters…” Can you explain that ?
Answer
The error counters M are initialized at a high value
After that all pattern-blocks are compared to the corresponding text piece and the counter value is updated to a lower value
In this particular case, when e = the error counter can get as low as k+1, which is higher than the initial value
jk /
Any other questions?