SORTINGLecture 11

CS2110 – Spring 2019

"Organizing is what you do before you do something, so that when you do it, it is not all mixed up."

~ A. A. Milne

Prelim 1: Tuesday, 12 March

Visit exams page of course website. It tells you your assigned time to take it (5:30 or 7:30) and what to do if you have a conflict.Anyone with any kind of a conflict must complete assignment

P1 Conflict

on the CMS by midnight of Wednesday, 6 March.It is extremely important that this be done correctly and on time. We have to schedule room and proctors and know how many of each prelim (5:30 or 7:30) to print.

2

Recitation next week

Review for prelim!

3

Why Sorting?

¨ Sorting is useful¤ Database indexing¤ Operations research¤ Compression

¨ There are lots of ways to sort¤ There isn't one right answer¤ You need to be able to figure out the options and decide

which one is right for your application.¤ Today, we'll learn several different algorithms

(and how to develop them)

4

We look at four sorting algorithms

¨ Insertion sort¨ Selection sort¨ Quick sort¨ Merge sort

5

InsertionSort

6

pre: b0 b.length

? post: b0 b.length

sorted

inv:

or: b[0..i-1] is sorted

b0 i b.length

sorted ?

A loop that processes elements of an array

in increasing orderhas this invariant ---just replace “sorted”

by “processed”.

inv:

or: b[0..i-1] is processed

b0 i b.lengthprocessed ?

Each iteration, i= i+1; How to keep inv true?

7

inv: b0 i b.length

sorted ?

b0 i b.length2 5 5 5 7 3 ? e.g.

b0 i b.length2 3 5 5 5 7 ?

88

inv: b0 i b.length

sorted ?

b0 i b.length2 5 5 5 7 3 ? e.g.

Push b[i] to its sorted position in b[0..i], then increase i

What to do in each iteration?

2 5 5 5 3 7 ?

2 5 5 3 5 7 ?

2 5 3 5 5 7 ?

2 3 5 5 5 7 ?

Loop body

(inv true before

and after)

b0 i b.length2 3 5 5 5 7 ?

This will take time proportional to the number of swaps needed

9

// sort b[], an array of int// inv: b[0..i-1] is sortedfor (int i= 0; i < b.length; i= i+1) {

// Push b[i] down to its sorted// position in b[0..i]

}

Note English statementin body.

Abstraction. Says whatto do, not how.

This is the best way to present it. We expectyou to present it this

way when asked.

Later, can show how toimplement that with an

inner loop

Insertion Sort

Present algorithm like this

Insertion Sort10

// sort b[], an array of int// inv: b[0..i-1] is sortedfor (int i= 0; i < b.length; i= i+1) {

// Push b[i] down to its sorted// position in b[0..i]

}

invariant P: b[0..i] is sortedexcept that b[k] may be < b[k-1]

while (k > 0 && b[k] < b[k-1]) {

}

start?stop?

progress?k= k–1;

maintaininvariant?

Swap b[k] and b[k-1];

int k= i;

2 5 3 5 5 7 ? ik

example

11

�Worst-case: O(n2)(reverse-sorted input)

�Best-case: O(n)(sorted input)

�Expected case: O(n2)

// sort b[], an array of int// inv: b[0..i-1] is sortedfor (int i= 0; i < b.length; i= i+1) {

// Push b[i] down to its sorted// position in b[0..i]}

Pushing b[i] down can take i swaps.Worst case takes

1 + 2 + 3 + … n-1 = (n-1)*n/2swaps.

Let n = b.length

Insertion Sort

12

// sort b[], an array of int// inv: b[0..i-1] is sortedfor (int i= 0; i < b.length; i= i+1) {

// Push b[i] down to its sorted// position in b[0..i]}

A sorting algorithm is stable if two equal values stay in the same relative position.initial: (3 7, 2 8, 7, 6)stably sorted (2, 3, 6, 7, 7, 8)unstably sorted (2, 3, 6, 7, 7, 8)

Let n = b.length

Insertion Sort is stable

Insertion sort is stable

Performance

Algorithm Ave time. Worst-case time Space Stable?

Insertion Sort ! "# . !("#) !(1) Yes

13

SelectionSort

14

pre: b0 b.length

? post: b0 b.lengthsorted

inv: b0 i b.lengthsorted , <= b[i..] >= b[0..i-1] Additional term

in invariant

Keep invariant true while making progress?

e.g.: b0 i b.length1 2 3 4 5 6 9 9 9 7 8 6 9

Increasing i by 1 keeps inv true only if b[i] is min of b[i..]

SelectionSort

15 Another common way for people to sort cards

Runtimewith n = b.length§Worst-case O(n2)§Best-case O(n2)§Expected-case O(n2)

// sort b[].// inv: b[0..i-1] sorted AND// b[0..i-1] <= b[i..]for (int i= 0; i < b.length; i= i+1) {

int m= index of min of b[i..];Swap b[i] and b[m];

}

sorted, smaller values larger valuesb0 i length

Each iteration, swap min value of this section into b[i]

SelectionSort —not stable

16 // sort b[].// inv: b[0..i-1] sorted AND// b[0..i-1] <= b[i..]for (int i= 0; i < b.length; i= i+1) {

int m= index of min of b[i..];Swap b[i] and b[m];

}

sorted, smaller values 8 7 8 3 5 6 b0 i length

Here, swapping b[i] with the minimum of b[i..], 3, changes the relative position of the two 8’s.

Performance17

Algorithm Ave time. Worst-case time Space Stable?

Insertion sort !(#$) !(#$) !(1) Yes

Selection sort ! #$ !(#$) !(1) No

QuickSort18

Quicksort developed by Sir Tony Hoare (he wasknighted by the Queen of England for hiscontributions to education and CS).84 years old.Developed Quicksort in 1958. But he could notexplain it to his colleague, so he gave up on it.Later, he saw a draft of the new language Algol 58 (which became Algol 60). It had recursive procedures. First time in a procedural programming language. “Ah!,” he said. “I know how to write it better now.” 15 minutes later, his colleague also understood it.

Dijkstras, Hoares, Grieses, 1980s19

Partition algorithm of quicksort20

Swap array values around until b[h..k] looks like this:

x ? h h+1 k

<= x x >= x h j k

pre:

post:

x is called the pivot

20 31 24 19 45 56 4 20 5 72 14 9921

pivotpartition

j19 4 5 14 20 31 24 45 56 20 72 99

Not yet sorted

Not yet sorted

these can be in any order

these can be in any order The 20 could

be in the other partition

Partition algorithm of quicksort

Partition algorithm

22

x ? h h+1 k

<= x x >= x h j k

b

b

<= x x ? >= x h j t k

b

pre:

post:

Combine pre and post to get an invariant

invariant needs at

least 4 sections

Partition algorithm23

<= x x ? >= x h j t k

b

j= h; t= k;while (j < t) {

if (b[j+1] <= b[j]) {Swap b[j+1] and b[j]; j= j+1;

} else {Swap b[j+1] and b[t]; t= t-1;

}}

Terminate when j = t, so the “?” segment is empty, so diagram looks like result diagram

Initially, with j = h and t = k, this diagram looks like the start diagram

Takes linear time: O(k+1-h)

/** Sort b[h..k]. */public static void QS(int[] b, int h, int k) {

if (b[h..k] has < 2 elements) return;

Function does the partition algorithm and returns position j of pivot

int j= partition(b, h, k);// We know b[h..j–1] <= b[j] <= b[j+1..k]

}

QuickSort procedure

24

Base case

// Sort b[h..j-1] and b[j+1..k]

QS(b, h, j-1); QS(b, j+1, k);

<= x x >= x h j k

Worst case quicksort: pivot always smallest value

25

x0 >= x0j n

partioning at depth 0

x0 x1 >= x1j

partioning at depth 1

x0 x1 x2 >= x2j

partioning at depth 2

/** Sort b[h..k]. */public static void QS(int[] b, int h, int k) {

if (b[h..k] has < 2 elements) return;int j= partition(b, h, k);QS(b, h, j-1); QS(b, j+1, k);

Depth of recursion: O(n)

Processing at depth i: O(n-i)

O(n*n)

Best case quicksort: pivot always middle value

26

≤ x0 x0 ≥ x00 j n

depth 0. 1 segment of size n to partition.

≤ x1 x1 ≥ x1 x0 ≤ x2 x2 ≥ x2 Depth 1. 2 segments of size ≤ n/2 to partition.Depth 2. 4 segments of size ≤ n/4 to partition.

Max depth: O(log n). Time to partition on each level: O(n)Total time: O(n log n).

Average time for Quicksort: n log n. Difficult calculation

QuickSort complexity to sort array of length n

27

/** Sort b[h..k]. */public static void QS(int[] b, int h, int k) {

if (b[h..k] has < 2 elements) return;int j= partition(b, h, k);// We know b[h..j–1] <= b[j] <= b[j+1..k]// Sort b[h..j-1] and b[j+1..k]QS(b, h, j-1); QS(b, j+1, k);

}

Time complexityWorst-case: O(n*n)Average-case: O(n log n)

Worst-case space: O(n)! --depth of recursion can be n

Can rewrite it to have space O(log n)Show this at end of lecture if we have time

Worst-case space: ?What’s depth of recursion?

Partition. Key issue. How to choose pivot28

Popular heuristics: Usew first array value (not so good)w middle array value (not so good)w Choose a random element (not so good)w median of first, middle, last, values (often used)!

x ? h k

<= x x >= x h j k

b

b

pre:

post:

Choosing pivotIdeal pivot: the median, since it splits array in halfBut computing the median is O(n), quite complicated

Performance29

Algorithm Ave time. Worst-case time Space Stable?

Insertion sort ! "# !("#) !(1) Yes

Selection sort ! "# !("#) !(1) No

Quick sort !(" log ") !("#) O(log n)* No

Merge sort

* The first algorithm we developed takes space O(n) in the worst case, but it can be reduced to O(log n)

Merge two adjacent sorted segments

/* Sort b[h..k]. Precondition: b[h..t] and b[t+1..k] are sorted. */public static merge(int[] b, int h, int t, int k) {

Copy b[h..t] into a new array c;Merge c and b[t+1..k] into b[h..k];

}

30

sorted sorted h t k

merged, sorted h k

Merge two adjacent sorted segments

/* Sort b[h..k]. Precondition: b[h..t] and b[t+1..k] are sorted. */public static merge(int[] b, int h, int t, int k) {}

31

4 7 7 8 9 3 4 7 8b

3 4 4 7 7 7 8 8 9b

h t k

sorted sorted h t k

merged, sorted h k

Merge two adjacent sorted segments32

? ? ? ? ? 3 4 7 8bh t k

4 7 7 8 9c0 t

Place all values in c[0..] andb[t+1..k] intob[h..k] in sorted order

3 4 4 7 7 7 8 8 9bh t k

Merge two adjacent sorted segments33

? ? ? ? ? 3 4 7 8bh t k

4 7 7 8 9c0

Place all values in c[0..] andb[t+1..k] intob[h..k] in sorted order

Merge two adjacent sorted segments34

3 ? ? ? ? ? 4 7 8bh t k

4 7 7 8 9c0

Step 1. Move 3from b[t+1] to b[h]

Merge two adjacent sorted segments35

3 4 ? ? ? ? 4 7 8bh t k

? 7 7 8 9c0

Step 2. Move 4from c[0] to b[h+1]

Merge two adjacent sorted segments36

3 4 4 ? ? ? ? 7 8bh t k

? 7 7 8 9c0

Step 3. Move 4from b[t+2] to b[h+2]

Merge two adjacent sorted segments37

3 4 4 7 ? ? ? 7 8bh t k

? ? 7 7 8 9c0

Step 4. Move 7from c[1] to b[h+3]

Invariant:

moved still to movec0 i

In place, sorted ? still to movebh u v k

if i < c.length and h < u, b[u-1] ≤ c[i]

if v ≤ k and h < u, b[u-1] ≤ b[v]

Merge two adjacent sorted segments

// Merge sorted c and b[t+1..k] into b[h..k]

38

xc

x and y, sorted

? ybh t k

head of x tail of xc0 i c.lengthinvariant:

0 t-h pre: x, y are sorted

post: bh k

tail of y? bh u v k

head of x and head of y, sorted

39 i = 0; u = h; v = t+1;while( i <= t-h){

if(v <= k && b[v] < c[i]) {b[u] = b[v]; u++; v++;

}else {b[u] = c[i]; u++; i++;

}}

}

Merge

sortedc

sorted

? sortedbh t k

sorted sortedc0 i c.lengthinv:

0 t-h pre:

post: bh k

sortedsorted ? bh u v k

Mergesort

/** Sort b[h..k] */public static void mergesort(int[] b, int h, int k]) {

if (size b[h..k] < 2) return;int t= (h+k)/2;mergesort(b, h, t);mergesort(b, t+1, k);merge(b, h, t, k);

}

40

h t k

merged, sorted h k

sorted sorted

41

QuickSort versus MergeSort41

/** Sort b[h..k] */public static void QS

(int[] b, int h, int k) {if (k – h < 1) return;int j= partition(b, h, k);QS(b, h, j-1); QS(b, j+1, k);

}

/** Sort b[h..k] */public static void MS

(int[] b, int h, int k) {if (k – h < 1) return;MS(b, h, (h+k)/2); MS(b, (h+k)/2 + 1, k);merge(b, h, (h+k)/2, k);

}

One processes the array then recurses.One recurses then processes the array.

Performance42

Algorithm Ave time. Worst-case time Space Stable?

Insertion sort !(#$). !(#$) !(1) Yes

Selection sort ! #$ . !(#$) !(1) No

Quick sort !(# log #) . !(#$) O(log n)* No

Merge sort !(# log #) . !(# log #) O(n) Yes

* The first algorithm we developed takes space O(n) in the worst case, but it can be reduced to O(log n)

Sorting in Java

¨ Java.util.Arrays has a method sort(array)¤ implemented as a collection of overloaded methods¤ for primitives, sort is implemented with a version of

quicksort¤ for Objects that implement Comparable, sort is

implemented with timSort, a modified mergesort developed in 1993 by Tim Peters

¨ Tradeoff between speed/space and stability/performance guarantees

43

Quicksort with logarithmic space

Problem is that if the pivot value is always the smallest (or always the largest), the depth of recursion is the size of the array to sort.

Eliminate this problem by doing some of it iteratively and some recursively. We may show you this later. Not today!

44

QuickSort with logarithmic space

45

/** Sort b[h..k]. */public static void QS(int[] b, int h, int k) {int h1= h; int k1= k;// invariant b[h..k] is sorted if b[h1..k1] is sortedwhile (b[h1..k1] has more than 1 element) {

Reduce the size of b[h1..k1], keeping inv true}

}

QuickSort with logarithmic space

46

/** Sort b[h..k]. */public static void QS(int[] b, int h, int k) {

int h1= h; int k1= k;// invariant b[h..k] is sorted if b[h1..k1] is sortedwhile (b[h1..k1] has more than 1 element) {

int j= partition(b, h1, k1);// b[h1..j-1] <= b[j] <= b[j+1..k1]if (b[h1..j-1] smaller than b[j+1..k1])

{ QS(b, h, j-1); h1= j+1; }else

{QS(b, j+1, k1); k1= j-1; }}

}

Only the smaller segment is sorted

recursively. If b[h1..k1] has size n, the smaller

segment has size < n/2.Therefore, depth of

recursion is at most log n