lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity,...
TRANSCRIPT
![Page 1: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/1.jpg)
lecture 5: complexity, data-structuresand sortingSTAT 545: Intro. to Computational Statistics
Vinayak RaoPurdue University
September 3, 2019
![Page 2: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/2.jpg)
Linear algebra
Let x and y be N× 1 vectors
Let A and B be N× N and N×M matrices
How many additions and multiplications to calculate:
• x⊤y• Ax• AB• A−1
1/15
![Page 3: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/3.jpg)
Big-O notation
The big-O notation provides an asymptotic upper bound:
O(g(N)) = {f : ∃c,N0 > 0 s.t. f(N) ≤ cg(N) ∀N > N0}
2N3 ∈ O(N3)N2 ∈ O(N3)N3 + N2 ∈ O(N3)N3 + exp(N) ̸∈ O(N3)
So is matrix multiplication O(N3)? Yes, but: it’s also O(N2.38)!
Conjecture: matrix multiplication is actually O(N2).
2/15
![Page 4: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/4.jpg)
Big-O notation
The big-O notation provides an asymptotic upper bound:
O(g(N)) = {f : ∃c,N0 > 0 s.t. f(N) ≤ cg(N) ∀N > N0}
2N3 ∈ O(N3)N2 ∈ O(N3)N3 + N2 ∈ O(N3)N3 + exp(N) ̸∈ O(N3)
So is matrix multiplication O(N3)? Yes, but: it’s also O(N2.38)!
Conjecture: matrix multiplication is actually O(N2).
2/15
![Page 5: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/5.jpg)
Big-O notation
The big-O notation provides an asymptotic upper bound:
O(g(N)) = {f : ∃c,N0 > 0 s.t. f(N) ≤ cg(N) ∀N > N0}
2N3 ∈ O(N3)N2 ∈ O(N3)N3 + N2 ∈ O(N3)N3 + exp(N) ̸∈ O(N3)
So is matrix multiplication O(N3)? Yes, but: it’s also O(N2.38)!
Conjecture: matrix multiplication is actually O(N2).
2/15
![Page 6: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/6.jpg)
Big-O notation
The big-O notation provides an asymptotic upper bound:
O(g(N)) = {f : ∃c,N0 > 0 s.t. f(N) ≤ cg(N) ∀N > N0}
2N3 ∈ O(N3)N2 ∈ O(N3)N3 + N2 ∈ O(N3)N3 + exp(N) ̸∈ O(N3)
So is matrix multiplication O(N3)? Yes, but: it’s also O(N2.38)!
Conjecture: matrix multiplication is actually O(N2).
2/15
![Page 7: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/7.jpg)
Sorting
Consider a set of N numbers. We want to sort them indecreasing order. What is the complexity?
Naïve algorithm:
• Find smallest number. Cost? O(N)• Find next smallest number. Cost? O(N)• · · ·
Overall cost? O(N2)Can we do better?
3/15
![Page 8: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/8.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
83
9
4/15
![Page 9: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/9.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
83
9
4/15
![Page 10: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/10.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
83
9
4/15
![Page 11: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/11.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
83
4/15
![Page 12: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/12.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
83
6
4/15
![Page 13: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/13.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
83
6
4/15
![Page 14: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/14.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12
8
36
4/15
![Page 15: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/15.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12 3
6
4/15
![Page 16: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/16.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12 3
6
4/15
![Page 17: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/17.jpg)
Priority queue
A ‘bag’ with three commands: Insert, FindMax and RemoveMax.
4 3 12 3
6
Sorting, clustering, discrete-event simulation, queuing systemsHow do we implement this?
4/15
![Page 18: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/18.jpg)
Priority queue
Naive approach 1: an unsorted array:
(3, 1, 4, 2, 6, 8, 3)
3 1 4 2 6 8 3
Start End
What is the cost of Insert? O(1)What is the cost of FindMax? O(N)What is the cost of RemoveMax (assume we’ve already found themaximum)? O(1)
5/15
![Page 19: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/19.jpg)
Priority queue
Naive approach 1: an unsorted array:
(3, 1, 4, 2, 6, 8, 3)
3 1 4 2 6 8 3
Start End
What is the cost of Insert? O(1)What is the cost of FindMax? O(N)What is the cost of RemoveMax (assume we’ve already found themaximum)? O(1)
5/15
![Page 20: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/20.jpg)
Priority queue
Naive approach 2: a sorted array:
(1, 2, 3, 3, 4, 6, 8)
31 42 6 83
Start End
Cost of FindMax? O(1)Cost of RemoveMax? O(1)Cost of Insert.FindPosition? O(log(N)) (binary search)Cost of Insert.Insert? O(N)
6/15
![Page 21: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/21.jpg)
Priority queue
Naive approach 2: a sorted array:
(1, 2, 3, 3, 4, 6, 8)
31 42 6 83
Start End
Cost of FindMax? O(1)Cost of RemoveMax? O(1)Cost of Insert.FindPosition? O(log(N)) (binary search)Cost of Insert.Insert? O(N)
6/15
![Page 22: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/22.jpg)
Priority queue
Naive approach 3: a sorted doubly linked-list:
28
n
n
n
n
n
n
n
31
Start 6
4
p
p
pp
p
p
p
3End
What is the cost of FindMax? O(1)What is the cost of RemoveMax? O(1)What is the cost of Insert? O(N)Each approach solves one problem, but makes anotheroperation N. Can we do better?
7/15
![Page 23: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/23.jpg)
Priority queue
Naive approach 3: a sorted doubly linked-list:
28
n
n
n
n
n
n
n
31
Start 6
4
p
p
pp
p
p
p
3End
What is the cost of FindMax? O(1)What is the cost of RemoveMax? O(1)What is the cost of Insert? O(N)
Each approach solves one problem, but makes anotheroperation N. Can we do better?
7/15
![Page 24: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/24.jpg)
Priority queue
Naive approach 3: a sorted doubly linked-list:
28
n
n
n
n
n
n
n
31
Start 6
4
p
p
pp
p
p
p
3End
What is the cost of FindMax? O(1)What is the cost of RemoveMax? O(1)What is the cost of Insert? O(N)Each approach solves one problem, but makes anotheroperation N. Can we do better? 7/15
![Page 25: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/25.jpg)
HeapsWe need a more complicated data-structure: a Heap.
2
2
4
1 1 3
7
8
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 26: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/26.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 27: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/27.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 28: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/28.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 29: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/29.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 30: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/30.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 31: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/31.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 32: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/32.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 33: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/33.jpg)
Heaps
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 34: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/34.jpg)
Heaps
2
2
4
1 1 3
7
8
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 35: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/35.jpg)
Heaps
2
4
2
1 1 3
7
8
For a precise definition, see: http://pages.cs.wisc.edu/~vernon/cs367/notes/11.PRIORITY-Q.html
8/15
![Page 36: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/36.jpg)
Heaps: Insert
2
2
4
1 1 3
7
8
9/15
![Page 37: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/37.jpg)
Heaps: Insert
2
2
4
1 1 3
7
8
9/15
![Page 38: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/38.jpg)
Heaps: Insert
2
2
4
1 1 3
7
8
9
9/15
![Page 39: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/39.jpg)
Heaps: Insert
2
4
1 1 3
7
8
2
9
9/15
![Page 40: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/40.jpg)
Heaps: Insert
2
1 1 3
7
8
2
4
9
9/15
![Page 41: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/41.jpg)
Heaps: Insert
2
1 1 3
7
2
4
8
9
9/15
![Page 42: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/42.jpg)
Heaps: Insert
2
1 1 3
7
2
4
8
9
Cost? O(log(N))
9/15
![Page 43: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/43.jpg)
Heaps: RemoveMax
2
1 1 3
7
2
4
8
9
10/15
![Page 44: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/44.jpg)
Heaps: RemoveMax
2
4
78
2
1 1 3
10/15
![Page 45: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/45.jpg)
Heaps: RemoveMax
2
4
7
8
2
1 1 3
Swap with larger child
10/15
![Page 46: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/46.jpg)
Heaps: RemoveMax
2
4 7
8
2
1 1 3
10/15
![Page 47: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/47.jpg)
Heaps: RemoveMax
2
4 7
8
2
1 1 3
Cost? O(log(N))
10/15
![Page 48: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/48.jpg)
Heapsort
Consider a set of N numbers. Want to sort in decreasing order.
Grow a priority queue, sequentially adding elements
• Cost of each step? O(log(N))• Overall cost? O(N log(N))
Sequentially remove the maximum element
• Cost of each step? O(log(N))• Overall cost? O(N log(N))
Cost of overall algorithm? O(N log(N))
11/15
![Page 49: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/49.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
7 3 2 1 5 9 4
12/15
![Page 50: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/50.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
7 3 2 1 5 9 4
• Pivot: #7 (Blue)• Start: #1 (Red)• End: #6
12/15
![Page 51: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/51.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
4 3 2 1 5 9 7
• Pivot: #1• Start: #2• End: #6
12/15
![Page 52: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/52.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
4 3 2 1 5 9 7
• Pivot: #1• Start: #2• End: #6
12/15
![Page 53: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/53.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
3 4 2 1 5 9 7
• Pivot: #2• Start: #3• End: #6
12/15
![Page 54: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/54.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
3 2 4 1 5 9 7
• Pivot: #3• Start: #4• End: #6
12/15
![Page 55: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/55.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
3 2 1 4 5 9 7
Recurse for each half
12/15
![Page 56: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/56.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
3 2 1 4 5 9 7
Recurse for each half
12/15
![Page 57: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/57.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
32 4 591 7
Recurse for each half
12/15
![Page 58: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/58.jpg)
Quicksort
See http://me.dt.in.th/page/Quicksort/ for a nicer display
• Choose a pivot• Ensure all elements to le t of pivot are less than/equal, andall to right are greater than the pivot
• Recurse for each half
32 4 591 7
At the end, we have a sorted list
12/15
![Page 59: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/59.jpg)
Analysis of quicksort
Analysis is a bit harder
What is the worst-case runtime?
What is the best-case runtime?
Average run-time is Θ(n log n)
Average with respect to what?
Randomized algorithms
13/15
![Page 60: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/60.jpg)
Final (informal) comments
Class P: Problems of polynomial complexity. Let T(n) berunning-time for input size n. Then there is a k such that:
T(n) = O(nk)
Class E: Problems of exponential complexity.
T(n) = O(exp(n))
Class NP: Problems of where proposed solution can be verifiedin polynomial time. E.g. graph isomorphismClass NP-complete: Hardest problems in NP (i.e. problems inboth NP and NP-hard). E.g. travelling salesman.Class NP-hard: at least as hard as the hardest problems in NP(halting problem)
14/15
![Page 61: lecture 5: complexity, data-structures and sorting · 2019-12-11 · lecture 5: complexity, data-structures and sorting STAT545:Intro.toComputationalStatistics VinayakRao PurdueUniversity](https://reader034.vdocuments.us/reader034/viewer/2022050308/5f70056d8fc9001344555893/html5/thumbnails/61.jpg)
P = NP?
A million dollar question (literally)
Com
ple
xit
yP ≠ NP P = NP
NP-Hard
NP-Complete
P
NP
NP-Hard
P = NP =NP-Complete
Figure: from Wikipedia
15/15