chapter 1 divide and conquer -...

Chapter 1

Divide and Conquer

Algorithm TheoryWS 2016/17

Fabian Kuhn

Algorithm Theory, WS 2016/17 Fabian Kuhn 2

Operations on Polynomials

Cost depending on representation:

Coefficient Roots Point-Value

Evaluation 𝑶(𝒏) 𝑶(𝒏) 𝑶 𝒏𝟐

Addition 𝑶 𝒏 ∞ 𝑶 𝒏

Multiplication 𝑶(𝒏𝟏.𝟓𝟖) 𝑶 𝒏 𝑶 𝒏


Coefficients to Point-Value Representation

Goal: Compute 𝑝 𝑥 for all 𝑥 in a given set 𝑋 of size 𝑋 = 𝑁

• Divide 𝑝 𝑥 of degr. 𝑁 − 1 into 2 polynomials of degr. Τ𝑁 2− 1

𝑝0 𝑦 = 𝑎0 + 𝑎2𝑦 + 𝑎4𝑦2 +⋯+ 𝑎𝑁−2𝑦

Τ𝑁 2−1 (even coeff.)

𝑝1 𝑦 = 𝑎1 + 𝑎3𝑦 + 𝑎5𝑦2 +⋯+ 𝑎𝑁−1𝑦

Τ𝑁 2−1 (odd coeff.)

Let’s first look at the “combine” step:

∀𝑥 ∈ 𝑋 ∶ 𝑝 𝑥 = 𝑝0 𝑥2 + 𝑥 ⋅ 𝑝1 𝑥2

• Recursively compute 𝑝0 𝑦 and 𝑝1(𝑦) for all 𝑦 ∈ 𝑋2

– Where 𝑋2 ≔ 𝑥2 ∶ 𝑥 ∈ 𝑋

• Generally, we have 𝑋2 = |𝑋|


Choice of 𝑋

• Select points 𝑥0, 𝑥1, … , 𝑥𝑁−1 to evaluate 𝑝 and 𝑞 in a clever way

Consider the 𝑵 complex roots of unity:

𝝎𝟖𝟏

𝝎𝟖𝟐

𝝎𝟖𝟑

𝝎𝟖𝟒

𝝎𝟖𝟓

𝝎𝟖𝟔

𝝎𝟖𝟕

𝝎𝟖𝟎 = 𝟏

Principle root of unity: 𝝎𝑵 = 𝒆 ൗ𝟐𝝅𝒊𝑵

𝑖 = −1, 𝑒2𝜋𝑖 = 1

Powers of 𝝎𝒏 (roots of unity):

1 = 𝜔𝑁0 , 𝜔𝑁

1 , … ,𝜔𝑁𝑁−1

Note: 𝜔𝑁𝑘 = 𝑒 ൗ2𝜋𝑖𝑘

𝑁 = cos2𝜋𝑘

𝑁+ 𝑖 ⋅ sin

2𝜋𝑘

𝑁


Properties of the Roots of Unity

• Cancellation Lemma:

For all integers 𝑛 > 0, 𝑘 ≥ 0, and 𝑑 > 0, we have:

𝝎𝒅𝒏𝒅𝒌 = 𝝎𝒏

𝒌 , 𝝎𝒏𝒌+𝒏 = 𝝎𝒏

𝒌

• Proof:


Properties of the Roots of Unity

Claim: If 𝑋 = 𝜔2𝑘𝑖 ∶ 𝑖 ∈ 0,… , 2𝑘 − 1 , we have

𝑿𝟐 = 𝝎𝒌𝒊 ∶ 𝒊 ∈ 𝟎,… , 𝒌 − 𝟏 , 𝑿𝟐 =

𝑿

𝟐


Analysis

New recurrence formula:

𝑇 𝑁, 𝑋 ≤ 𝑇 ൗ𝑁 2 , ൗ𝑋2 + 𝑂(𝑁 + 𝑋 )


Faster Polynomial Multiplication?

Idea to compute 𝑝 𝑥 ⋅ 𝑞(𝑥) (for polynomials of degree < 𝑛):

𝑝, 𝑞 of degree 𝑛 − 1, 𝑛 coefficients

2 × 2𝑛 point-value pairs 𝜔2𝑛𝑘 , 𝑝 𝜔2𝑛

𝑘 and 𝜔2𝑛𝑘 , 𝑞 𝜔2𝑛

𝑘

2𝑛 point-value pairs 𝜔2𝑛𝑘 , 𝑝 𝜔2𝑛

𝑘 𝑞 𝜔2𝑛𝑘

𝑝 𝑥 𝑞(𝑥) of degree 2𝑛 − 2, 2𝑛 − 1 coefficients

Evaluation at points 𝜔2𝑛0 , 𝜔2𝑛

1 , … , 𝜔2𝑛2𝑛−1

Point-wise multiplication

Interpolation


Discrete Fourier Transform

• The values 𝑝 𝜔𝑁𝑖 for 𝑖 = 0,… ,𝑁 − 1 uniquely define a

polynomial 𝑝 of degree < 𝑁.

Discrete Fourier Transform (DFT):

• Assume 𝑎 = 𝑎0, … , 𝑎𝑁−1 is the coefficient vector of poly. 𝑝

𝑝 𝑥 = 𝑎𝑁−1𝑥𝑁−1 +⋯+ 𝑎1𝑥 + 𝑎0

DFT𝑁 𝑎 ≔ 𝑝 𝜔𝑁0 , 𝑝 𝜔𝑁

1 , … , 𝑝 𝜔𝑁𝑁−1


Example

• Consider polynomial 𝑝 𝑥 = 3𝑥3 − 15𝑥2 + 18𝑥

• 𝑁 = 4, roots of unity: 𝜔40 = 1,𝜔4

1 = 𝑖, 𝜔42 = −1, 𝜔4

3 = −𝑖

• Evaluate 𝑝(𝑥) at 𝜔4𝑘:

𝜔40, 𝑝 𝜔4

0 = 1, 𝑝 1 = 1,6

𝜔41, 𝑝 𝜔4

1 = 𝑖, 𝑝 𝑖 = 𝑖, 15 + 15𝑖

𝜔42, 𝑝 𝜔4

2 = −1, 𝑝 −1 = −1,−36

𝜔43, 𝑝 𝜔4

3 = −𝑖, 𝑝 −𝑖 = −𝑖, 15 − 15𝑖

• For 𝑎 = 0,18,−15,3 :

𝐃𝐅𝐓𝟒 𝒂 = (𝟔, 𝟏𝟓 + 𝟏𝟓𝒊,−𝟑𝟔, 𝟏𝟓 − 𝟏𝟓𝒊)


DFT: Recursive Structure

Evaluation for 𝑘 = 0,… ,𝑁 − 1:

𝑝 𝜔𝑁𝑘 = 𝑝0 (𝜔𝑁

𝑘 )2 +𝜔𝑁𝑘 ⋅ 𝑝1 (𝜔𝑁

𝑘 )2

= ൞𝑝0 𝜔 Τ𝑁 2

𝑘 +𝜔𝑁𝑘 ⋅ 𝑝1 𝜔 Τ𝑁 2

𝑘 if 𝑘 < ൗ𝑁 2

𝑝0 𝜔 Τ𝑁 2𝑘− Τ𝑁 2 +𝜔𝑁

𝑘 ⋅ 𝑝1 𝜔 Τ𝑁 2𝑘− Τ𝑁 2 if 𝑘 ≥ ൗ𝑁 2

For the coefficient vector 𝑎 of 𝑝 𝑥 :

DFT𝑁 𝑎 = 𝑝0 𝜔 Τ𝑁 20 , … , 𝑝0 𝜔 Τ𝑁 2

Τ𝑁 2−1 , 𝑝0 𝜔 Τ𝑁 20 , … , 𝑝0 𝜔 Τ𝑁 2

Τ𝑁 2−1

+ 𝜔𝑁0 𝑝0 𝜔 Τ𝑁 2

0 , … , 𝜔𝑁Τ𝑁 2−1

𝑝0 𝜔 Τ𝑁 2Τ𝑁 2−1

, 𝜔𝑁Τ𝑁 2𝑝0 𝜔 Τ𝑁 2

0 , … , 𝜔𝑁𝑁−1𝑝0 𝜔 Τ𝑁 2

Τ𝑁 2−1


Example

For the coefficient vector 𝑎 of 𝑝 𝑥 :

DFT𝑁 𝑎 = 𝑝0 𝜔 Τ𝑁 20 , … , 𝑝0 𝜔 Τ𝑁 2

Τ𝑁 2−1 , 𝑝0 𝜔 Τ𝑁 20 , … , 𝑝0 𝜔 Τ𝑁 2

Τ𝑁 2−1

+ 𝜔𝑁0 𝑝0 𝜔 Τ𝑁 2

0 , … , 𝜔𝑁Τ𝑁 2−1

𝑝0 𝜔 Τ𝑁 2Τ𝑁 2−1

, 𝜔𝑁Τ𝑁 2𝑝0 𝜔 Τ𝑁 2

0 , … , 𝜔𝑁𝑁−1𝑝0 𝜔 Τ𝑁 2

Τ𝑁 2−1

𝑁 = 4:

𝑝 𝜔40 = 𝑝0 𝜔2

0 +𝜔40𝑝1 𝜔2

0

𝑝 𝜔41 = 𝑝0 𝜔2

1 +𝜔41𝑝1 𝜔2

1

𝑝 𝜔42 = 𝑝0 𝜔2

0 +𝜔42𝑝1 𝜔2

0

𝑝 𝜔43 = 𝑝0 𝜔2

1 +𝜔43𝑝1 𝜔2

1

Need: 𝑝0 𝜔20 , 𝑝0 𝜔2

1 and 𝑝1 𝜔20 , 𝑝1 𝜔2

1

(DFTs of coefficient vectors of 𝑝0 and 𝑝1)


Summary: Computation of DFT𝑁

• Divide-and-conquer algorithm for DFT𝑁(𝑝):

1. Divide

𝑁 ≤ 1: DFT1 𝑝 = 𝑎0

𝑁 > 1: Divide 𝑝 into 𝑝0 (even coeff.) and 𝑝1 (odd coeff).

2. Conquer

Solve DFT Τ𝑁 2(𝑝0) and DFT Τ𝑁 2(𝑝1) recursively

3. Combine

Compute DFT𝑁(𝑝) based on DFT Τ𝑁 2(𝑝0) and DFT Τ𝑁 2 𝑝1


Small Improvement

Polynomial 𝑝 of degree 𝑁 − 1:

𝑝 𝜔𝑁𝑘 = ൞

𝑝0 𝜔 Τ𝑁 2𝑘 +𝜔𝑁

𝑘 ⋅ 𝑝1 𝜔 Τ𝑁 2𝑘 if 𝑘 < ൗ𝑁 2

𝑝0 𝜔 Τ𝑁 2𝑘− Τ𝑁 2 +𝜔𝑁

𝑘 ⋅ 𝑝1 𝜔 Τ𝑁 2𝑘− Τ𝑁 2 if 𝑘 ≥ ൗ𝑁 2

= ൞𝑝0 𝜔 Τ𝑁 2

𝑘 + 𝜔𝑁𝑘 ⋅ 𝑝1 𝜔 Τ𝑁 2

𝑘 if 𝑘 < ൗ𝑁 2

𝑝0 𝜔 Τ𝑁 2𝑘− Τ𝑁 2 −𝜔𝑁

𝑘− Τ𝑁 2 ⋅ 𝑝1 𝜔 Τ𝑁 2𝑘− Τ𝑁 2 if 𝑘 ≥ ൗ𝑁 2

Need to compute 𝑝0 𝜔 Τ𝑁 2𝑘 and 𝜔𝑁

𝑘 ⋅ 𝑝1 𝜔 Τ𝑁 2𝑘 for 0 ≤ 𝑘 < Τ𝑁 2.


Example 𝑁 = 8

𝑝 𝜔80 = 𝑝0 𝜔4

0 +𝜔80 ⋅ 𝑝1 𝜔4

0

𝑝 𝜔81 = 𝑝0 𝜔4

1 + 𝜔81 ⋅ 𝑝1(𝜔4

1)

𝑝 𝜔82 = 𝑝0 𝜔4

2 + 𝜔82 ⋅ 𝑝1(𝜔4

2)

𝑝 𝜔83 = 𝑝0 𝜔4

3 + 𝜔83 ⋅ 𝑝1(𝜔4

3)

𝑝 𝜔84 = 𝑝0 𝜔4

0 − 𝜔80 ⋅ 𝑝1(𝜔4

0)

𝑝 𝜔85 = 𝑝0 𝜔4

1 − 𝜔81 ⋅ 𝑝1(𝜔4

1)

𝑝 𝜔86 = 𝑝0 𝜔4

2 − 𝜔82 ⋅ 𝑝1(𝜔4

2)

𝑝 𝜔87 = 𝑝0 𝜔4

3 − 𝜔83 ⋅ 𝑝1(𝜔4

3)


Fast Fourier Transform (FFT) Algorithm

Algorithm FFT(a)

• Input: Array 𝑎 of length 𝑁, where 𝑁 is a power of 2

• Output: DFT𝑁 𝑎

if 𝑛 = 1 then return 𝑎0; // 𝑎 = 𝑎0𝑑 0 ≔ FFT 𝑎0, 𝑎2, … , 𝑎𝑁−2 ;

𝑑 1 ≔ FFT 𝑎1, 𝑎3, … , 𝑎𝑁−1 ;

𝜔𝑁 ≔ 𝑒 ൗ2𝜋𝑖𝑁; 𝜔 ≔ 1;

for 𝑘 = 0 to Τ𝑁 2− 1 do // 𝜔 = 𝜔𝑁𝑘

𝑥 ≔ 𝜔 ⋅ 𝑑𝑘1

;

𝑑𝑘 ≔ 𝑑𝑘0+ 𝑥; 𝑑𝑘+ Τ𝑁 2 ≔ 𝑑𝑘

0− 𝑥;

𝜔 ≔ 𝜔 ⋅ 𝜔𝑁

end;

return 𝑑 = [𝑑0, 𝑑1, … , 𝑑𝑁−1];


Example

𝑝 𝑥 = 3𝑥3 − 15𝑥2 + 18𝑥 + 0, 𝑎 = [0,18,−15,3]







𝑘




Evaluation at 𝜔2𝑛0 , 𝜔2𝑛

1 , … , 𝜔2𝑛2𝑛−1 using FFT


Interpolation


Interpolation

Convert point-value representation into coefficient representation

Input: 𝑥0, 𝑦0 , … , 𝑥𝑛−1, 𝑦𝑛−1 with 𝑥𝑖 ≠ 𝑥𝑗 for 𝑖 ≠ 𝑗

Output:

Degree-(𝑛 − 1) polynomial with coefficients 𝑎0, … , 𝑎𝑛−1 such that

𝑝 𝑥0 = 𝑎0 + 𝑎1𝑥0 + 𝑎2𝑥02 +⋯+ 𝑎𝑛−1𝑥0

𝑛−1 = 𝑦0𝑝 𝑥1 = 𝑎0 + 𝑎1𝑥1 + 𝑎2𝑥1

2 +⋯+ 𝑎𝑛−1𝑥1𝑛−1 = 𝑦1

⋮ ⋮𝑝 𝑥𝑛−1 = 𝑎0 + 𝑎1𝑥𝑛−1 + 𝑎2𝑥𝑛−1

2 +⋯+ 𝑎𝑛−1𝑥𝑛−1𝑛−1 = 𝑦𝑛−1

linear system of equations for 𝑎0, … , 𝑎𝑛−1


Interpolation

Matrix Notation:

1 𝑥0 ⋯ 𝑥0𝑛−1

1 𝑥1 ⋯ 𝑥1𝑛−1

⋮ ⋮ ⋱ ⋮1 𝑥𝑛−1 ⋯ 𝑥𝑛−1

𝑛−1

⋅

𝑎0𝑎1⋮

𝑎𝑛−1

=

𝑦0𝑦1⋮

𝑦𝑛−1

• System of equations solvable iff 𝑥𝑖 ≠ 𝑥𝑗 for all 𝑖 ≠ 𝑗

Special Case 𝒙𝒊 = 𝝎𝒏𝒊 :

1 1 1 ⋯ 11 𝜔𝑛 𝜔𝑛

2 ⋯ 𝜔𝑛𝑛−1

1 𝜔𝑛2 𝜔𝑛

4 ⋯ 𝜔𝑛2(𝑛−1)

⋮ ⋮ ⋮ ⋱ ⋮

1 𝜔𝑛𝑛−1 𝜔𝑛

2 𝑛−1⋯ 𝜔𝑛

(𝑛−1)(𝑛−1)

⋅

𝑎0𝑎1𝑎2⋮

𝑎𝑛−1

=

𝑦0𝑦1𝑦2⋮

𝑦𝑛−1


Interpolation

• Linear system:𝑊 ⋅ 𝒂 = 𝒚 ⟹ 𝒂 = 𝑊−1 ⋅ 𝒚

𝑊𝑖,𝑗 = 𝜔𝑛𝑖𝑗, 𝒂 =

𝑎0⋮

𝑎𝑛−1, 𝒚 =

𝑦0⋮

𝑦𝑛−1

Claim:

𝑾𝒊𝒋−𝟏 =

𝝎𝒏−𝒊𝒋

𝒏

Proof: Need to show that 𝑊−1𝑊 = 𝐼𝑛


DFT Matrix Inverse

𝑊−1𝑊 =

⋯

1

𝑛

𝜔𝑛−𝑖

𝑛⋯

𝜔𝑛− 𝑛−1 𝑖

𝑛⋮⋯

⋅

⋯ 1 ⋯

⋯ 𝜔𝑛𝑗

⋯

⋯ 𝜔𝑛2𝑗

⋯⋮

⋯ 𝜔𝑛𝑛−1 𝑗

⋯


DFT Matrix Inverse

𝑊−1𝑊 𝑖,𝑗 =

ℓ=0

𝑛−1𝜔𝑛ℓ(𝑗−𝑖)

𝑛

Need to show 𝑊−1𝑊 𝑖,𝑗 = ቊ1 if 𝑖 = 𝑗0 if 𝑖 ≠ 𝑗

Case 𝒊 = 𝒋:


DFT Matrix Inverse

𝑊−1𝑊 𝑖,𝑗 =

ℓ=0

𝑛−1𝜔𝑛ℓ(𝑗−𝑖)

𝑛

Case 𝒊 ≠ 𝒋:


Inverse DFT

• 𝑊−1 =

⋯1

𝑛

𝜔𝑛−𝑘

𝑛⋯

𝜔𝑛− 𝑛−1 𝑘

𝑛

⋮⋯

• We get 𝒂 = 𝑊−1 ⋅ 𝒚 and therefore

𝑎𝑘 =1

𝑛

𝜔𝑛−𝑘

𝑛⋯

𝜔𝑛− 𝑛−1 𝑘

𝑛⋅

𝑦0𝑦1⋮

𝑦𝑛−1

=1

𝑛⋅

𝑗=0

𝑛−1

𝜔𝑛−𝑘𝑗

⋅ 𝑦𝑗


DFT and Inverse DFT

Inverse DFT:

𝑎𝑘 =1

𝑛⋅

𝑗=0

𝑛−1

𝜔𝑛−𝑘𝑗

⋅ 𝑦𝑗

• Define polynomial 𝑞 𝑥 = 𝑦0 + 𝑦1𝑥 +⋯+ 𝑦𝑛−1𝑥𝑛−1:

𝑎𝑘 =1

𝑛⋅ 𝑞(𝜔𝑛

−𝑘)

DFT:

• Polynomial 𝑝 𝑥 = 𝑎0 + 𝑎1𝑥 +⋯+ 𝑎𝑛−1𝑥𝑛−1:

𝑦𝑘 = 𝑝(𝜔𝑛𝑘)


DFT and Inverse DFT

𝑞 𝑥 = 𝑦0 + 𝑦1𝑥 +⋯+ 𝑦𝑛−1𝑥𝑛−1, 𝑎𝑘 =

1

𝑛⋅ 𝑞 𝜔𝑛

−𝑘 :

• Therefore:𝑎0, 𝑎1, … , 𝑎𝑛−1

=1


−0 , 𝑞 𝜔𝑛−1 , 𝑞 𝜔𝑛

−2 , … , 𝑞 𝜔𝑛− 𝑛−1

=1


0 , 𝑞 𝜔𝑛𝑛−1 , 𝑞 𝜔𝑛

𝑛−2 , … , 𝑞 𝜔𝑛1

• Recall:

DFT𝑛 𝒚 = 𝑞 𝜔𝑛0 , 𝑞 𝜔𝑛

1 , 𝑞 𝜔𝑛2 , … , 𝑞 𝜔𝑛

𝑛−1

= 𝑛 ⋅ (𝑎0, 𝑎𝑛−1, 𝑎𝑛−2, … , 𝑎2, 𝑎1)


DFT and Inverse DFT

• We have DFT𝑛(𝒚) = 𝑛 ⋅ (𝑎0, 𝑎𝑛−1, 𝑎𝑛−2, … , 𝑎2, 𝑎1):

𝑎𝑖 =

1

𝑛⋅ DFT𝑛(𝒚) 0 if 𝑖 = 0

1

𝑛⋅ DFT𝑛(𝒚) 𝑛−𝑖 if 𝑖 ≠ 0

• DFT and inverse DFT can both be computed using FFT algorithm in 𝑂 𝑛 log 𝑛 time.

• 2 polynomials of degr. < 𝑛 can be multiplied in time 𝑂(𝑛 log 𝑛).







𝑘




Evaluation at 𝜔2𝑛0 , 𝜔2𝑛

1 , … , 𝜔2𝑛2𝑛−1 using FFT


Interpolation using FFT


Convolution

• More generally, the polynomial multiplication algorithm computes the convolution of two vectors:

𝒂 = 𝑎0, 𝑎1, … , 𝑎𝑚−1

𝒃 = 𝑏0, 𝑏1, … , 𝑏𝑛−1

𝒂 ∗ 𝒃 = 𝑐0, 𝑐1, … , 𝑐𝑚+𝑛−2 ,

where 𝑐𝑘= (𝑖,𝑗):𝑖+𝑗=𝑘𝑖<𝑚,𝑗<𝑛

𝑎𝑖𝑏𝑗

• 𝑐𝑘 is exactly the coefficient of 𝑥𝑘 in the product polynomial of the polynomials defined by the coefficient vectors 𝒂 and 𝒃


More Applications of Convolutions

Signal Processing Example:

• Assume 𝒂 = 𝑎0, … , 𝑎𝑛−1 represents a sequence of measurements over time

• Measurements might be noisy and have to be smoothed out

• Replace 𝑎𝑖 by weighted average of nearby last 𝑚 and next 𝑚measurements (e.g., Gaussian smoothing):

𝑎𝑖′ =

1

𝑍⋅

𝑗=𝑖−𝑚

𝑖+𝑚

𝑎𝑗𝑒− 𝑖−𝑗 2

• New vector 𝒂′ is the convolution of 𝒂 and the weight vector1

𝑍⋅ 𝑒−𝑚

2, 𝑒− 𝑚−1 2

, … , 𝑒−1, 1, 𝑒−1, … , 𝑒− 𝑚−1 2, 𝑒−𝑚

2

• Might need to take care of boundary points…


More Applications of Convolutions

Combining Histograms:

• Vectors 𝒂 and 𝒃 represent two histograms

• E.g., annual income of all men & annual income of all women

• Goal: Get new histogram 𝒄 representing combined income of all possible pairs of men and women:

𝒄 = 𝒂 ∗ 𝒃

Also, the DFT (and thus the FFT alg.) has many other applications!


DFT in Signal Processing

Assume that 𝑦 0 , 𝑦 1 , 𝑦 2 ,… , 𝑦(𝑇 − 1) are measurements of a time-dependent signal.

Inverse DFT𝑁 of 𝑦 0 ,… , 𝑦 𝑇 − 1 is a vector 𝑐0, … , 𝑐𝑁−1 s.t.

𝑦 𝑡 =

𝑘=0

𝑁−1

𝑐𝑘 ⋅ 𝑒2𝜋𝑖⋅𝑘𝑁 ⋅𝑡

=

𝑘=0

𝑇−1

𝑐𝑘 ⋅ cos2𝜋 ⋅ 𝑘

𝑁⋅ 𝑡 + 𝑖 sin

2𝜋 ⋅ 𝑘

𝑁⋅ 𝑡

• Converts signal from time domain to frequency domain

• Signal can then be edited in the frequency domain– e.g., setting some 𝑐𝑘 = 0 filters out some frequencies

chapter 1 divide and conquer -...

Documents