ics 353: design and analysis of algorithms
DESCRIPTION
King Fahd University of Petroleum & Minerals Information & Computer Science Department. ICS 353: Design and Analysis of Algorithms. Dynamic Programming. Reading Assignment. M. Alsuwaiyel, Introduction to Algorithms: Design Techniques and Analysis , World Scientific Publishing Co., Inc. 1999. - PowerPoint PPT PresentationTRANSCRIPT
ICS 353: Design and Analysis of Algorithms
Dynamic Programming
King Fahd University of Petroleum & MineralsKing Fahd University of Petroleum & MineralsInformation & Computer Science DepartmentInformation & Computer Science Department
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Reading Assignment• M. Alsuwaiyel, Introduction to Algorithms:
Design Techniques and Analysis, World Scientific Publishing Co., Inc. 1999.• Chapter 7 Sections 1 - 4 and 6.
2
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Dynamic Programming• Dynamic Programming algorithms address
problems whose solution is recursive in nature, but has the following property: The direct implementation of the recursive solution results in identical recursive calls that are executed more than once.
• Dynamic programming implements such algorithms by evaluating the recurrence in a bottom-up manner, saving intermediate results that are later used in computing the desired solution
3
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Fibonacci Numbers•
• What is the recursive algorithm that computes Fibonacci numbers? What is its time complexity?• Note that it can be shown that
210
21
1
0
nfffff
nnn
251,2
nnf
4
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Computing the Binomial Coefficient• Recursive Definition
• Actual Value
nk
kn
kn
nkk
kn
01
11
or01
!!!
knkn
kn
5
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Computing the Binomial Coefficient• What is the direct recursive algorithm for
computing the binomial coefficient? How much does it cost?• Note that
nnnn
nn n
2
!2/!2/!
2/
6
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Optimization Problems and Dynamic Programming• Optimization problems with certain properties make
another class of problems that can be solved more efficiently using dynamic programming.
• Development of a dynamic programming solution to an optimization problem involves four steps
• Characterize the structure of an optimal solution • Optimal substructures, where an optimal solution consists of sub-
solutions that are optimal.• Overlapping sub-problems where the space of sub-problems is small in
the sense that the algorithm solves the same sub-problems over and over rather than generating new sub-problems.
• Recursively define the value of an optimal solution.• Compute the value of an optimal solution in a bottom-up
manner.• Construct an optimal solution from the computed optimal value.
7
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Longest Common Subsequence Problem• Problem Definition: Given two strings A and B
over alphabet , determine the length of the longest subsequence that is common in A and B.
• A subsequence of A=a1a2…an is a string of the form ai1ai2…aik where 1i1<i2<…<ik n
• Example: Let = { x , y , z }, A = xyxyxxzy, B=yxyyzxy, and C= zzyyxyz
• LCS(A,B)=yxyzy Hence the length =• LCS(B,C)= Hence the length =• LCS(A,C)= Hence the length =
8
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Straight-Forward Solution• Brute-force search
• How many subsequences exist in a string of length n?
• How much time needed to check a string whether it is a subsequence of another string of length m?
• What is the time complexity of the brute-force search algorithm of finding the length of the longest common subsequence of two strings of sizes n and m?
9
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Dynamic Programming Solution
• Let L[i,j] denote the length of the longest common subsequence of a1a2…ai and b1b2…bj, which are substrings of A and B of lengths n and m, respectively. ThenL[i,j] = when i = 0 or j = 0L[i,j] = when i > 0, j > 0, ai=bj
L[i,j] = when i > 0, j > 0, aibj
10
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
LCS AlgorithmAlgorithm LCS(A,B)Input: A and B strings of length n and m respectivelyOutput: Length of longest common subsequence of A and BInitialize L[i,0] and L[0,j] to zero;for i ← 1 to n do for j ← 1 to m do if ai = bj then
L[i,j] ← 1 + L[i-1,j-1] else L[i,j] ← max(L[i-1,j],L[i,j-1]) end if end for;end for;return L[n,m];
11
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Example (Q7.5 pp. 220)
• Find the length of the longest common subsequence of A=xzyzzyx and B=zxyyzxz
12
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Example (Cont.)x z y z z y x
0 0 0 0 0 0 0 0
z 0
x 0
y 0
y 0
z 0
x 0
z 0
13
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Complexity Analysis of LCS Algorithm• What is the time and space complexity of the
algorithm?
14
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Matrix Chain Multiplication• Assume Matrices A, B, and C have dimensions
210, 102, and 210 respectively. The number of scalar multiplications using the standard Matrix multiplication algorithm for• (A B) C is• A (B C) is
• Problem Statement: Find the order of multiplying n matrices in which the number of scalar multiplications is minimum.
15
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Straight-Forward Solution• Again, let us consider the brute-force method. We
need to compute the number of different ways that we can parenthesize the product of n matrices.• e.g. how many different orderings do we have for the
product of four matrices?• Let f(n) denote the number of ways to parenthesize the
product M1, M2, …, Mn.• (M1M2…Mk) (M k+1M k+2…Mn)
• What is f(2), f(3) and f(1)?
16
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Catalan Numbers
•
• Cn=f(n+1)
• Using Stirling’s Formula, it can be shown that f(n) is approximately
1221)(
nn
nnf
5.144
n
n
17
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Cost of Brute Force Method
• How many possibilities do we have for parenthesizing n matrices?
• How much does it cost to find the number of scalar multiplications for one parenthesized expression?
• Therefore, the total cost is
18
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
The Recursive Solution• Since the number of columns of each matrix Mi is equal
to the number of rows of Mi+1, we only need to specify the number of rows of all the matrices, plus the number of columns of the last matrix, r1, r2, …, rn+1 respectively.
• Let the cost of multiplying the chain Mi…Mj (denoted by Mi,j) be C[i,j]
• If k is an index between i+1 and j, what is the cost of multiplying Mi,j considering multiplying Mi,k-1 with Mk,j?
• Therefore, C[1,n]=
19
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
The Dynamic Programming AlgorithmC[1,1] C[1,2] C[1,3] C[1,4] C[1,5] C[1,6]
C[2,2] C[2,3] C[2,4] C[2,5] C[2,6]
C[3,3] C[3,4] C[3,5] C[3,6]
C[4,4] C[4,5] C[4,6]
C[5,5] C[5,6]
C[6,6]
20
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Example (Q7.11 pp. 221-222)• Given as input 2 , 3 , 6 , 4 , 2 , 7 compute the
minimum number of scalar multiplications:
21
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Example (Q7.11 pp. 221-222)0
M1
0M2
0M3
0M4
0M5
22
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
MatChain AlgorithmAlgorithm MatChainInput: r[1..n+1] of +ve integers corresponding to the dimensions
of a chain of matricesOutput: Least number of scalar multiplications required to
multiply the n matrices for i := 1 to n do C[i,i] := 0; // diagonal d0
for d := 1 to n-1 do // for diagonals d1 to dn-1
for i := 1 to n-d do j := i+d; C[i,j] := ; for k := i+1 to j do C[i,j] := min{C[i,j],C[i,k-1]+C[k,j]+r[i]r[k]r[j+1]; end for; end for; return C[1,n];
23
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Time and Space Complexity of MatChain Algorithm• Time Complexity
• Space Complexity
)(6
12121
)(1
1
3
1
1
2
1
11
1
11
1
1
11
1
111
1
1
n
nnncnncnddnc
dndcdcdc
cc
n
d
n
d
dn
i
n
d
dn
i
n
d
di
ik
dn
i
n
d
j
ik
dn
i
n
d
24
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
The Knapsack Problem• Let U = {u1, u2, …, un} be a set of n items to
be packed in a knapsack of size C.• Let sj and vj be the size and value of the jth
item, where sj, vj , 1 j n.• The objective is to fill the knapsack with some
items from U whose total size does not exceed C and whose total value is maximum.• Assume that the size of each item does not exceed
C.
25
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
The Knapsack Problem Formulation• Given n +ve integers in U, we want to find a
subset SU s.t.
is maximized subject to the constraint
Su
ii
v
CsSu
ii
26
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Inductive Solution• Let V[i,j] denote the value obtained by filling a
knapsack of size j with items taken from the first i items {u1, u2, …, ui} in an optimal way:• The range of i is • The range of j is• The objective is to find V[ , ]
• V[i,0] = V[0,j] =
• V[i,j] = V[i-1,j] if = max {V[i-1,j], V[ , ]+vi} if
27
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Example (pp. 223 Question 7.22)• There are five items of sizes 3, 5, 7, 8, and 9 with values 4,
6, 7, 9, and 10 respectively. The size of the knapsack is 22.
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
0 0 0
0 0 0
0 0 0
0 0 0
28
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Algorithm KnapsackAlgorithm KnapsackInput: A set of items U = {u1,u2,…,un} with sizes s1,s2,…,sn and
values v1,v2,…,vn, respectively and knapsack capacity C.
Output: the maximum value of subject to for i := 0 to n do V[i,0] := 0; for j := 0 to C do V[0,j] := 0; for i := 1 to n do for j := 1 to C do V[i,j] := V[i-1,j]; if si j then
V[i,j] := max{V[i,j], V[i-1,j-si]+vi}
end for; end for; return V[n,C];
Su
ii
v CsSu
ii
29
Dynamic Programming092 ICS 353: Design and Analysis of Algorithms
Time and Space Complexity of the Knapsack Algorithm
30