lecture 14

CS 473ug: Algorithms

Chandra [email protected]

3228 Siebel Center

University of Illinois, Urbana-Champaign

Fall 2007

Chekuri CS473ug

Longest Increasing Subsequence

Part I


Chekuri CS473ug



Sequence: an ordered list a1, a2, . . . , an. Length of sequence is n

ai1 , ai2 , . . . , aik is a subsequence of a1, a2, . . . , an if1 ≤ i1 < i2 < . . . < ik ≤ n.

A sequence is increasing if a1 < a2 < . . . < an. It is non-decreasingif a1 ≤ a2 ≤ . . . ≤ an. Similarly decreasing and non-increasing.


Input A sequence of numbers a1, a2, . . . , an

Goal Find an increasing subsequence ai1 , ai2 , . . . , aik ofmaximum length

Chekuri CS473ug


Example

Chekuri CS473ug


A First Recursive Approach

L(i): length of longest common subsequence in first i elementsa1, a2, . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i − 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i − 1)

Case 2 : L(i) contains ai , then L(i) =? What is the element in thesubsequence before ai? If it is aj then it better be the casethat aj < ai since we are looking for an increasing sequence.Do we know which j? No! So we try all possibilities

L(i) = 1 + maxj<i and aj<ai

L(j)

Is the above correct? No, because we do not know that L(j) isa subsequence that actually ends at aj !

Chekuri CS473ug






Case 2 : L(i) contains ai , then L(i) =?

What is the element in thesubsequence before ai? If it is aj then it better be the casethat aj < ai since we are looking for an increasing sequence.Do we know which j? No! So we try all possibilities


L(j)


Chekuri CS473ug






Case 2 : L(i) contains ai , then L(i) =? What is the element in thesubsequence before ai? If it is aj then it better be the casethat aj < ai since we are looking for an increasing sequence.

Do we know which j? No! So we try all possibilities


L(j)


Chekuri CS473ug






Case 2 : L(i) contains ai , then L(i) =? What is the element in thesubsequence before ai? If it is aj then it better be the casethat aj < ai since we are looking for an increasing sequence.Do we know which j? No!

So we try all possibilities


L(j)


Chekuri CS473ug








L(j)


Chekuri CS473ug








L(j)

Is the above correct?

No, because we do not know that L(j) isa subsequence that actually ends at aj !

Chekuri CS473ug








L(j)


Chekuri CS473ug


A Correct Recursion

L(i): longest common subsequence in first i elements a1, a2, . . . , ai

that ends in ai


L(j)

What is the final solution? maxni=1 L(i)

How many subproblems? O(n)

Chekuri CS473ug


A Correct Recursion


that ends in ai


L(j)

What is the final solution?

maxni=1 L(i)


Chekuri CS473ug


A Correct Recursion


that ends in ai


L(j)


How many subproblems?

O(n)

Chekuri CS473ug


A Correct Recursion


that ends in ai


L(j)



Chekuri CS473ug


Table based Iterative Algorithm

Recurrence:L(i) = 1 + max

j<i and aj<ai

L(j)

Iterative algorithm:

for i = 1 to n do

L[i ] = 1for j = 1 to i − 1 do

if aj < ai and 1 + L[j ] > L[i ]L[i ] = 1 + L[j ]

Output maxni=1 L[i ]

Running Time:

O(n2)Space: O(n)

Chekuri CS473ug




j<i and aj<ai

L(j)


for i = 1 to n do

L[i ] = 1for j = 1 to i − 1 do



Running Time: O(n2)Space:

O(n)

Chekuri CS473ug




j<i and aj<ai

L(j)


for i = 1 to n do

L[i ] = 1for j = 1 to i − 1 do



Running Time: O(n2)Space: O(n)

Chekuri CS473ug


A DAG Based Approach

Given sequence a1, a2, . . . , an create DAG as follows:

for each i there is a node vi

if i < j and ai < aj add an edge (vi , vj)

find longest path

Chekuri CS473ug


Dynamic Programming Methods

There is always a DAG of computation underlying every dynamicprogram but it is not always clear.

Three ways to come up with dynamic programming solutions

recursion followed by memoization with a polynomial numberof subproblems

bottom up via tables

identify a DAG followed by a shortest/longest path in DAG

But there is always a recursion+memoization that is implicit in theother methods!

Chekuri CS473ug


Dilworth’s Theorem

Given sequence a1, a2, . . . , an the longest increasing subsequencecan be of size 1.

Example: a1 > a2 > a3 > . . . > an

In the above example the longest decreasing sequence is of size n.

Question: Given a sequence of length n, can the longest increasingsequence and the longest decreasing sequence be both small?

Dilworth’s Theorem Consequence: In any sequence of length n,either the longest increasing sequence or the longest decresingsequence is of length b

√nc+ 1.

Exercise: give a sequence of length n in which both the longestincreasing subsequence and longest decreasing susbequence are nomore than b

√nc+ 1.

Chekuri CS473ug


Dilworth’s Theorem

Given sequence a1, a2, . . . , an the longest increasing subsequencecan be of size 1.Example: a1 > a2 > a3 > . . . > an

In the above example the longest decreasing sequence is of size n.

Question: Given a sequence of length n, can the longest increasingsequence and the longest decreasing sequence be both small?

Dilworth’s Theorem Consequence: In any sequence of length n,either the longest increasing sequence or the longest decresingsequence is of length b

√nc+ 1.

Exercise: give a sequence of length n in which both the longestincreasing subsequence and longest decreasing susbequence are nomore than b

√nc+ 1.

Chekuri CS473ug

Edit Distance

Part II

Edit Distance

Chekuri CS473ug

Edit Distance

Spell Checking Problem

Given a string “exponen” that is not in the dictionary, how shoulda spell checker suggest a nearby string?

What does nearness mean?

Question: Given two strings x1x2 . . . xn and y1y2 . . . ym what is adistance between them?

Edit Distance: minimum number of “edits” to transform x into y .

Chekuri CS473ug

Edit Distance

Edit Distance

Edit Distance: minimum number of “edits” to transform x into y .

Edit operations:

delete a letter

add a letter

substitute a letter with another letter

Why is substitute not “delete plus add”?

In general different edits can have different costs and usingsubstitution as a edit allows a single operation as far as distance isconcerned

Chekuri CS473ug

Edit Distance

Example

Chekuri CS473ug

Edit Distance

Edit Distance as Alignment

Chekuri CS473ug

Edit Distance

Edit Distance Problem

Input Two strings x = x1x2 . . . xn and y = y1y2 . . . ym oversome fixed alphabet Σ

Goal Find edit distance between x and y : minimumnumber of edits to transform x into y

Note: EditDist(x,y) = EditDist(y,x)

Chekuri CS473ug

Edit Distance

Towards a Recursive Solution

Think of aligning the strings. Can we express the full problem as afunction of subproblems?

Case 1 xn is aligned with ym. Then x1x2 . . . xn−1 is aligned withy1y2 . . . ym−1

Case 2a xn is aligned with and ym is to left of . Then x1x2 . . . xn−1

is aligned with y1y2 . . . yn.

Case 2b ym is aligned with and xn is to left of . Then x1x2 . . . xn isaligned with y1y2 . . . ym−1.

Subproblems involve aligning prefixes of the two strings.Find edit distance between prefix x [1..i ] of x and prefix y [1..j ] of yEditDist(x,y) is the distance between x [1..n] and y [1..m]

Chekuri CS473ug

Edit Distance

A Recursive Solution

E [i , j ]: edit distance between x [1..i ] and y [1..j ]

Case 1 xi aligned with xj .

E [i , j ] = diff (xi , yj) + E [i − 1, j − 1]

where diff (xi , yj) = 0 if xi = yj , otherwise diff (xi , xj) = 1

Case 2a xi is mapped to and xi is to right of xj

E [i , j ] = 1 + E [i − 1, j ]

Case 2b yj is mapped to and xi is to left of yj

E [i , j ] = 1 + E [i , j − 1]

Chekuri CS473ug

Edit Distance


E [i , j ] = min{diff (xi , yj)+E [i−1, j−1], 1+E [i−1, j ], 1+E [i , j−1]

Base cases:

E [i , 0] = i for i ≥ 0

E [0, j ] = j for i ≥ 0

How many subproblems? O(mn)

Chekuri CS473ug

Edit Distance



Base cases:E [i , 0] = i for i ≥ 0

E [0, j ] = j for i ≥ 0

How many subproblems?

O(mn)

Chekuri CS473ug

Edit Distance



Base cases:E [i , 0] = i for i ≥ 0

E [0, j ] = j for i ≥ 0

How many subproblems? O(mn)

Chekuri CS473ug

Edit Distance

Iterative Solution

What is table?

E is a two-dimensional array of size (n + 1)(m + 1)How do we order the computation?To compute E [i , j ] need to have computedE [i − 1, j − 1],E [i − 1, j ],E [i , j − 1].

Chekuri CS473ug

Edit Distance

Iterative Solution

What is table? E is a two-dimensional array of size (n + 1)(m + 1)How do we order the computation?

To compute E [i , j ] need to have computedE [i − 1, j − 1],E [i − 1, j ],E [i , j − 1].

Chekuri CS473ug

Edit Distance

Iterative Solution

What is table? E is a two-dimensional array of size (n + 1)(m + 1)How do we order the computation?To compute E [i , j ] need to have computedE [i − 1, j − 1],E [i − 1, j ],E [i , j − 1].

Chekuri CS473ug

Edit Distance

Iterative Solution

for i = 0 to nE [i , 0] = i

for j = 0 to mE [0, j ] = j

for i = 1 to n do

for j = 1 to m do

E [i , j ] = min{diff (xi , yj) + E [i , j ], 1 + E [i − 1, j ], 1 + E [i , j − 1]}

Running Time: O(nm)Space: O(nm)Can reduce space to O(n + m) if only distance is needed but notobvious how to actually compute the edits. Can do it with a cleverscheme (see Jeff’s notes).

Chekuri CS473ug

Edit Distance

Iterative Solution

for i = 0 to nE [i , 0] = i

for j = 0 to mE [0, j ] = j

for i = 1 to n do

for j = 1 to m do


Running Time:

O(nm)Space: O(nm)Can reduce space to O(n + m) if only distance is needed but notobvious how to actually compute the edits. Can do it with a cleverscheme (see Jeff’s notes).

Chekuri CS473ug

Edit Distance

Iterative Solution

for i = 0 to nE [i , 0] = i

for j = 0 to mE [0, j ] = j

for i = 1 to n do

for j = 1 to m do


Running Time: O(nm)Space:

O(nm)Can reduce space to O(n + m) if only distance is needed but notobvious how to actually compute the edits. Can do it with a cleverscheme (see Jeff’s notes).

Chekuri CS473ug

Edit Distance

Iterative Solution

for i = 0 to nE [i , 0] = i

for j = 0 to mE [0, j ] = j

for i = 1 to n do

for j = 1 to m do


Running Time: O(nm)Space: O(nm)

Can reduce space to O(n + m) if only distance is needed but notobvious how to actually compute the edits. Can do it with a cleverscheme (see Jeff’s notes).

Chekuri CS473ug

Edit Distance

Iterative Solution

for i = 0 to nE [i , 0] = i

for j = 0 to mE [0, j ] = j

for i = 1 to n do

for j = 1 to m do


Running Time: O(nm)Space: O(nm)Can reduce space to O(n + m) if only distance is needed but notobvious how to actually compute the edits. Can do it with a cleverscheme (see Jeff’s notes).

Chekuri CS473ug

Edit Distance

Example

Chekuri CS473ug

Edit Distance

Where is the DAG?

one node for each (i , j), 1 = 0 ≤ i ≤ n, 0 ≤ j ≤ m.

Edges for node (i , j): from (i − 1, j − 1) of cost diff (xi , yj),from (i − 1, j) of cost 1, from (i , j − 1) of cost 1

find shortest path from (0, 0) to (n,m)

Chekuri CS473ug

lecture 14

Documents

increasing sequence

increasing subsequence

casethat aj ai

subsequence of a1

decreasingif a1 a2

length of sequence

sequence of numbers

common subsequence