speeding up algorithms for hidden markov models by exploiting repetitions
DESCRIPTION
Speeding Up Algorithms for Hidden Markov Models by Exploiting Repetitions. Shay Mozes Oren Weimann (MIT) Michal Ziv-Ukelson (Tel-Aviv U.). Shortly:. Hidden Markov Models are extensively used to model processes in many fields - PowerPoint PPT PresentationTRANSCRIPT
Speeding Up Algorithms for Hidden Markov Models by Exploiting
Repetitions
Shay MozesOren Weimann (MIT)
Michal Ziv-Ukelson (Tel-Aviv U.)
2
Shortly:• Hidden Markov Models are extensively used to model
processes in many fields• The runtime of HMM algorithms is usually linear in
the length of the input• We show how to exploit repetitions to obtain speedup• First provable speedup of Viterbi’s algorithm• Can use different compression schemes• Applies to several decoding and training algorithms
3
Markov Models
q1 q2• statesq1 , … , qk
• transition probabilitiesPi←j
• emission probabilitiesei(σ) σєΣ
• time independent, discrete, finite
e1(A) = 0.3
e1(C) = 0.2
e1(G) = 0.2
e1(T) = 0.3
e2(A) = 0.2
e2(C) = 0.3
e2(G) = 0.3
e2(T) = 0.2
P1←1 = 0.9 P2←1 = 0.1 P2←2 = 0.8
P1←2 = 0.2
4
Hidden Markov Models1
k
2
k
1
2
1
k
time
states
observed string
2
k
1
2
x1 x2 xnx3
Markov Models
• We are only given the description of the model and the observed string
• Decoding: find the hidden sequence of states that is most likely to have generated the observed string
5
probability of best sequence of states that emits first 5 chars and ends in state 2
v6[4]= e4(c)·P4←2·v5[2]
probability of best sequence of states that emits first 5 chars and ends in state j
v6[4]= P4←2·v5[2]v6[4]= v5[2]v6[4]=maxj{e4(c)·P4←j·v5[j]}v5[2]
Decoding – Viterbi’s Algorithm1 2 3 4 5 6 7 8 9 n
1
2
3
4
5
6
a a c g a c g g t
states
time
6
Outline
• Overview• Exploiting repetitions• Using LZ78• Using Run-Length Encoding• Summary of results
7
vn=M(xn) ⊗M(xn-1) ⊗ ··· ⊗M(x1) ⊗ v0
v2 = M(x2) ⊗M(x1) ⊗ v0
VA in Matrix Notation
Viterbi’s algorithm:
v1[i]=maxj{ei(x1)·Pi←j · v0[j]}v1[i]=maxj{ Mij(x1) · v0[j]}
Mij(σ) = ei (σ)·Pi←j
v1 = M(x1) ⊗ v0
(A⊗B)ij= maxk{Aik ·Bkj }
O(k2n)
O(k3n)
8
• use it twice!
vn=M(W)⊗M(t)⊗M(W)⊗M(t)⊗M(a)⊗M(c) ⊗v0
Exploiting Repetitionsc a t g a a c t g a a c
12 steps
6 steps
vn=M(c)⊗M(a)⊗M(a)⊗M(g)⊗M(t)⊗M(c)⊗M(a)⊗M(a)⊗M(g)⊗M(t)⊗M(a)⊗M(c)⊗v0
• compute M(W) = M(c)⊗M(a)⊗M(a)⊗M(g) once
9
ℓ - length of repetition W
λ – number of times W repeats in string
computing M(W) costs (ℓ -1)k3
each time W appears we save (ℓ -1)k2
W is good if λ(ℓ -1)k2 > (ℓ -1)k3
number of repeats λ > k number of states
Exploiting repetitions
>
matrix-matrix multiplication
matrix-vector multiplication
10
I. dictionary selection:choose the set D={Wi } of good substrings
II. encoding:compute M(Wi ) for every Wi in D
III. parsing:partition the input X into good substringsX = Wi1
Wi2 … Win’
X’ = i1,i2, … ,in’
IV. propagation:run Viterbi’s Algorithm on X’ using M(Wi)
General Scheme
Offline
11
Outline
• Overview• Exploiting repetitions• Using LZ78• Using Run-Length Encoding• Summary of results
12
LZ78
• The next LZ-word is the longest LZ-word previously seen plus one character
• Use a triea
c
g
g
aacgacg
• Number of LZ-words is asymptotically < n ∕ log n
13
I. O(n)
II. O(k3n ∕ log n)
III. O(n)
IV. O(k2n ∕ log n)
Using LZ78Cost
I. dictionary selection:D = words in LZ parse of X
II. encoding: use incremental nature of LZM(Wσ)= M(W) ⊗ M(σ)
III. parsing:X’ = LZ parse of X
IV. propagation:run VA on X’ using M(Wi )
Speedup: k2n log n
k3n ∕ log n k
14
• Remember speedup condition: λ > k • Use just LZ-words that appear more than k times• These words are represented by trie nodes with more
than k descendants• Now must parse X (step III) differently• Ensures graceful degradation with increasing k:
Speedup: min(1,log n ∕ k)
Improvementa
c
g
g
15
Experimental results
• Short - 1.5Mbp chromosome 4 of S. Cerevisiae (yeast)• Long - 22Mbp human Y-chromosome
~x5 faster:
16
Outline
• Overview• Exploiting repetitions• Using LZ78• Using Run-Length Encoding• Summary of results
17
Run Length Encodingaaaccggggg → a3c2g5
aaaccggggg → a2a1c2g4g1
18
Summary of results• General framework • LZ78 log(n) ∕ k• RLE r ∕ log(r)• Byte-Pair Encoding r• Path reconstruction O(n)• F/B algorithms (standard matrix multiplication)• Viterbi training same speedups apply• Baum-Welch training speedup, many details• Parallelization
19
Thank you!
Any questions?
20
Path traceback
• In VA, easy to do in O(n) time by keeping track of maximizing states during computation
• The problem: we run VA on X’, so we get the sequence of states for X’, not for X.we only get the states on the boundaries of good substrings of X
• Solution: keep track of maximizing states when computing the matrices M(w). Takes O(n) time and O(nk2) space
21
Training
• Estimate unknown parameters Pi←j , ei(σ)• Use Expectation Maximization:
1. Decoding2. Recalculate parameters
• Viterbi Training: each iteration costs O( VA + n + k2)
Decoding (bottleneck) speedup!
path traceback +
update Pi←j , ei(σ)
22
Baum Welch Training
• each iteration costs: O( FB + nk2)
• If substring w has length l and repeats λ times satisfies:
then can speed up the entire process by precalculation
2
2
kllk
path traceback +
update Pi←j , ei(σ)
Decoding O(nk2)