sorting chapter 7 sorting - william & marytadavis/cs303/ch07sm.pdf · chapter 7 sorting...
TRANSCRIPT
Chapter 7Sorting
Introduction
2
sortingfundamental task in data managementwell-studied problem in computer science
basic problemgiven an of items where each item contains akey, rearrange the items so that the keys appear inascending orderthe key may be only of the item begin sorted
e.g., the item could be an entire block of informationabout a student, while the search key might be only thestudent's name
Introduction
3
we will assumethe array to sort contains only integersdefined < and > operators (comparison-based sorting)all array positions contain data to be sortedthe entire sort can be performed in _________________
number of elements is relatively small: < a few milliongiven a list of n items, we will use thenotation to indicate that the search for adoes not follow that ofsorts that cannot be performed in main memory
may be performed on disk or tapeknown as sorting
Sorting Algorithms
4
sortsinsertion sortselection sortbubble sortwhy ?
Shellsort: ___________________sorts
heapsortmergesortquicksorta lower bound on sorting by _ comparison
sorting algorithms: count sortstring sorts
Sorting
5
given keys to sort, there are possible ofthe keys!
e.g, given and the keys , there arepossible permutations:
brute force enumeration of permutations is notcomputationally once
Sorting
6
cost model for sortingthe basic operations we will count are and___________if there are array accesses that are not associated withcomparisons or swaps, we need to count them, too
programming notesif the objects being sorted are large, we should swap______________ to the objects, rather than the objectsthemselvespolymorphism: general-purpose sorting algorithms vstemplated algorithms
Insertion Sort
7
insertion sortsimple
passesin pass , , we move to its correctlocation amongfor passes to , insertion sort ensures that theelements in positions to are in sorted orderat the end of pass , the elements in positions to are insorted order
Insertion Sort
8
algorithm
for (p = 1; p < N; p++) {tmp =for (j = p; (j > 0) && (tmp < ); j--) {
swap and}
= tmp}
Insertion Sort
9
to avoid the full swap, we can use the following code:
for (p = 1; p < N; p++) {tmp =for (j = p; (j > 0) && (tmp < ); j--) {
=}
= tmp}
Insertion Sort
10
example
Insertion Sort
11
exampleposition 0 1 2 3 4 5initial sequence 42 6 1 54 0 7p = 1 6 42 1 54 0 7p = 2 6 1 42 54 0 7
1 6 42 54 0 7p = 3 1 6 42 54 0 7p = 4 1 6 42 0 54 7
1 6 0 42 54 71 0 6 42 54 70 1 6 42 54 7
p = 5 0 1 6 42 7 540 1 6 7 42 54
Insertion Sort
12
analysisbest case: the keys are alreadycomparisons, no swapsworst case: the keys are in sorted orderexpected (average) case: ?
Insertion Sort
13
analysisgiven that we wish to sort in ascending order, an_______________ is any pair that is out of order relative toone another: for which butthe list
contains the following inversions:
swapping an adjacent pair of elements that are out of order______________ exactly one inversion
thus, any sort that operates by swapping adjacent termsrequires as many swaps as there are inversions
Insertion Sort
14
number of inversionsa list can have between and inversions, thelatter of which occurs whenthus, counting inversions also says the worst-case behaviorof insertion is ________________
what do inversions tell us about the behavior?let be the probability space of all permutations ofdistinct elements with equal _____________________
Theorem: The expected number of inversions in a listtaken from isthus, the expected complexity of insertion sort isquadratic
Insertion Sort
15
proofobserve that any pair in a list that is an inversion is in thecorrect order in the of that list
this means that if we look at a list and its reverse and countthe total number of inversions, then the combined numberof inversions is
list inversions reverse lists inversions1, 2, 3, 4 0 4, 3, 2, 1 62, 1, 3, 4 1 4, 3, 1, 2 53, 2, 1, 4 3 4, 1, 2, 3 3
Insertion Sort
16
proof (cont.)since there are distinct pairs of lists and theirreverses, there is a total of
inversions among the possible lists of distinct objects
this means that the expected number of inversions in anygiven list is
Insertion Sort
17
in summaryfor randomly ordered arrays of length with distinct keys,insertion sort uses
comparisons and swaps on ___________comparisons and swaps in the _________
casecomparisons and swaps in the case
Selection Sort
18
selection sortfind the smallest item and exchange it with the first entryfind the next smallest item and exchange it with the secondentryfind the next smallest item and exchange it with the thirdentry
Selection Sort
19
algorithm
for (p = 0; p < N; p++) {m = pfor (j = p+1; j < 0; j++) {
if ( < ) {m = j
}}swap and
}
Selection Sort
20
example
position 0 1 2 3 4initial sequence 42 6 9 54 0after p = 0 0 42 9 54 6after p = 1 0 6 9 54 42after p = 2 0 6 9 54 42after p = 3 0 6 9 42 54
Selection Sort
21
complexitycomparisons and swaps to sort an array of
lengththe amount of work is of the inputselection sort is no faster on input than onrandom inputselection sort involves a smaller number of thanany of the other sorting algorithms we will consider
Selection Sort
22
proofthere is one swap per iteration of the loop,which is executed timesthere is one comparison made in each iteration of the_______________ loop, which is executed
Bubble Sort
23
bubble sortread the items from left to rightif two items are out of order, swap themrepeat until sorteda sweep with swaps means we are done
Bubble Sort
24
algorithm
not_done = truewhile (not_done) {
not_done = falsefor (i = 0 to N - 2) {
if ( > ) {swap andnot_done = true
}}
}
Bubble Sort
25
example
initial sequence 42 6 9 54 0while-loop 6 42 9 54 0
6 9 42 54 06 9 42 0 54
while-loop 6 9 0 42 54while-loop 6 0 9 42 54while-loop 0 6 9 42 54
Bubble Sort
26
complexityif the data are already sorted, we make only sweepthrough the listotherwise, the complexity depends on the number of timeswe execute the while-loopsince bubble sort swaps items, it will have_______________ worst-case and expected-casecomplexity
Shellsort
27
ShellsortDonald L. Shell (1959), A high-speed sorting procedure,Communications of the ACM 2 (7): 3032.swapping only adjacent items dooms us to quadratic worst-case behavior, so swap items!Shellsort starts with an sequence
it uses insertion sort to sortevery -th term starting at , then thenevery -th term starting at , then thenetc.every term starting at , after which the array issorted
Shellsort
28
Shellsortsuppose we use the increment sequence 15, 7, 5, 3, 1, andhave finished the 15-sort and 7-sortthen we know that
we also know that
putting these together, we see that
Shellsort
29
Shellsortafter we have performed the sort using increment , thearray is -sorted: all elements that are terms apart arein the order:
the key to Shellsort's is the following fact:an -sorted array remains -sorted after sorting withincrement
Shellsort
30
example
Shellsort
31
complexitya good increment sequence has theproperty that for any element , when it is time for the -sort, there are only a few elements to the left of that are___________ thanShell's original increment sequence has ____________behavior:
,
Shellsort
32
complexity (cont.)the sequences
(T. H. Hibbard, 1963)
(V. R. Pratt, 1971)
yield worst-case complexityother sequences yield worst-case complexity
Heapsort
33
priority queues can be used to sort instrategy
build binary of elements timeperform deleteMin operationselements (smallest first) stored in second array, then____________ back into original array time
requires extra , which doubles memoryrequirement
Heapsort
34
can avoid extra array by storing element in original arrayheap leaves an as each smallest elementis deletedstore element in newly opened space, which is no longerused by the heapresults in list of order in arrayto achieve increasing order, change ordering of heap to________ heapwhen complete, array contains elements in ascendingorder
Heapsort
35
max heap after buildHeap phase
Heapsort
36
max heap after first deleteMax
Heapsort
37
analysisbuilding binary heap of elements < comparisonstotal deleteMax operations ifheapsort in worst caseaverage case extremely complex to compute
improved to or simplyheapsort useful if we want to sort the or____________ elements and
Mergesort
38
mergesort is a algorithmin Vol. III of The Art of Computer Programming, Knuthattributes the algorithm to John von Neumann (1945)the idea of mergesort is simple:
divide the array in twosort each half
the two subarrays using mergesortmerging simple since sorted
mergesort can be implemented recursively and non-recursivelyruns in , worst case
Mergesort
39
merging algorithm takes two arrays, and , and _________array
also uses three counters: , ,initialized to beginning of respective arrays
smaller of and copied towhen either input array exhausted, the remainder of theother list is to
Mergesort
40
example
Mergesort
41
example (cont.)
Mergesort
42
example (cont.)another way to visualize
Mergesort
43
time to merge two lists is at mostcomparisons
every comparison adds an element tomergesort easy to characterize
if , only one element to sortotherwise, mergesort first half andsecond halfmerge these two halvesproblem is divided into smaller problems and solvedrecursively, and conquered by patching the solutionstogether
Mergesort
44
analysisrunning time represented by relationassume is a power of 2 so that list is always divided____________for , time to mergesort is _________________otherwise, time to mergesort numbers is time to performtwo recursive mergesort of size , plus the time tomerge, which is linear
Mergesort
45
analysis (cont.)standard recurrence relationcan be solved in at least two ways
telescoping divide the recurrence through bysubstitution
Mergesort
46
solving mergesort recurrence relation using ______________
Mergesort
47
solving mergesort recurrence relation using telescoping(cont.)
Mergesort
48
solving mergesort recurrence relation using ______________(cont.)
Mergesort
49
proof by (cont.)
Prove: is equivalent to
letBase: (or )
rr: by definitioncf:
I.H.: Assume
for some
Mergesort
50
proof by induction (cont.)
I.S.: Show:
by I.H.
By induction, we have therefore shown the original statementto be true.
Quicksort
51
historically, quicksort has been known genericsorting algorithmaverage running timeworst case running time , but can be made highly________________can be combined with to achieveaverage and worst case time
Quicksort
52
quicksort is a divide-and-conquer algorithmbasic idea
arbitrarily select a single itemform three groups:
those than the itemthose to the itemthose than the item
recursively sort the first and third groupsconcatenate the three groups
Quicksort
53
Quicksort
54
implementation performance good on most inputsif list contains many , performance is verygoodsome issues
making extra lists recursively consumes ______________not much better than mergesort
loop bodies too heavycan avoid category in loop
Quicksort
55
classic quicksort algorithm to sort an arrayif there are either 0 or 1 elements in , then ____________choose an element in to serve as the ____________partition into two disjoint subsets and with theproperties that
if andif
apply quicksort recursively to andnote the for elements equal to the pivot
ideally, half of the duplicates would go into each sublist
Quicksort
56
examplepivot chosen randomly
Quicksort
57
many methods for selecting pivot and partitioning elementsvery sensitive to even slight variances
in these choicescomparison with mergesort
like mergesort, recursively solves two subproblems andrequires additional workunlike mergesort, may not be of equalsize (bad)
Quicksort
58
quicksort vs. mergesortmergesort: is trivial; the work is in the mergequicksort: the work is in the partitioning; the is trivalmergesort: requires an auxiliary array to be efficient (in-place variants exist that are less efficient, or which sacrificean important property called stability)quicksort: faster since partitioning step can be performedefficiently in place (with a modest amount ( ) spaceneeded to handle the recursion)in both sorts, more efficient to switch to insertion sort oncethe arrays are sufficiently small to avoid the cost of theoverhead of on small arrays
Quicksort
59
examplepivots in red
Quicksort
60
choosing the pivotpopular, but bad, method: choose the element inthe list
OK if input is _______________not OK if input is presorted or in reverse order
happens consistently in recursive callsresults in time for presorted datafor doing nothing!occurs oftenalternative: choose larger of first two elements
could pick the pivot randomlysafe, but random number generation expensive
Quicksort
61
choosing the pivot (cont.)median-of-three partitioning
best choice would be median of sublist, but takes toolong to calculategood estimate by picking three elements andusing middle element as pivot
randomness not really helpfulselect first, middle, and last elements
eliminates bad case for inputreduces number of by about 15%
example: 8, 1, 4, 9, 6, 3, 5, 2, 7, 0from 8, 0, and , or 6, select 6
Quicksort
62
partitioning strategyfirst, get pivot out of the way by swapping with __________elementtwo counters, and
starts at first elementstarts at nest-to-last element
move all the smaller elements to and all largerelements to __________
Quicksort
63
partitioning strategy (cont.)while is to the left of
move right, skipping over elements smaller than pivotmove left, skipping over elements larger than pivot
when and stop, is at a element and isat a element themexample
Quicksort
64
partitioning strategy (cont.)example (cont.)
after and cross, swap location with pivot
Quicksort
65
partitioning strategy (cont.)at this point, all positions contain smaller elementsthan , and all positions contain larger elementshow to handle equal elements
should stop when element to pivot? what about ?and should behave similarly to avoid all elements
equal to pivot collecting in one sublistbest to have and stop and perform an unnecessaryswap to avoid sublists (and quadratic run time!)
for small arrays, and as sublists get small (< 20 elements),use insertion sort
fast and avoids degenerate median-of-three cases
Quicksort
66
implementationdriver
pass array and range (left and right) to be sorted
Quicksort
67
implementation (cont.)median-of-three pivot selection
sort a[left], a[right], and a[center] in ________smallest of three ends up in locationlargest in last locationpivot in a[right 1]
can be initialized to left + 1can be initialized to right 2
since a[left] smaller than pivot, it will act as a____________ and stop from going past the beginningof the arraystoring pivot at a[right 1] will act as a sentinel for
Quicksort
68
implementation (cont.)median-of-three
Quicksort
69
implementation (cont.)main quicksort
Quicksort
70
implementation (cont.)main quicksort (cont.)
Quicksort
71
implementation (cont.)main quicksort (cont.)
16: i and j start at one off22: swap can be written inline19-20: small inner loop very fast
Quicksort
72
analysisquicksort is interesting because its worst-case behaviorand its expected behavior are very _________________let be the run-time needed to sort items
pivot is constant timecost of the partition isif has elements, then has elements,and
Quicksort
73
worst-case analysisthe worst-case occurs when or i.e., whenthe is the smallest or largest element everytime quicksort() is calledin this case, without loss of generality we may assume that
, so
thus
Quicksort
74
worst-case analysis (cont.)combining these yields
or
quadratic!is it likely that at every recursive call to quicksort() wewill choose the smallest element as the pivot?
yes, if the data are already sorted
Quicksort
75
best-case analysisin the best case, the pivot is always the of thedata being operated on
we know from the analysis of mergesort that the solution is
Quicksort
76
average-case analysisassumption: any size is equally likelyfor instance, suppose ; since we remove the ,the possible sizes of the partitions are
in this case the value of is
Quicksort
77
Quicksort
78
Quicksort
79
Lower Bound for Pairwise Sorting
80
no algorithm based on comparisons can guaranteesorting items with fewer than comparisons
to show this, we first abstract the behavior of suchalgorithms using a treea decision tree is a binary tree in which each noderepresents a set of possible orderingsthe root consists of the possible orderings of the items tobe sortedthe edges represent the results of comparisons, and anode comprises the orderings consistent with thecomparisons made on the path from the root to the nodeeach consists of a single sorted ordering
Lower Bound for Pairwise Sorting
81
Lower Bound for Pairwise Sorting
82
a decision tree to sort items must have leavesthis requires a tree of depth byapproximationthus, the best case for sorting with pairwise comparisons is
Quickselect
83
thus far, the best performance to select the smallestelement is using a priority queue (heap)quicksort can be modified to solve the selection problem
quickselect
Quickselect
84
quickselect algorithmgiven a set , let be its cardinalityquickselect
if there is 1 element in , returnchoose an element in to serve as the pivotpartition into two disjoint subsets and with theproperties that
if andif
now the search proceeds on andif , then the smallest element must be in , soquickselect( , )if , then the pivot is the smallest, so returnotherwise, the smallest element must be in , and it isthe -th element of , so return quickselect( ,
)
Quickselect
85
example: find the median of 7 items ( )red denotes pivots, while grey denotes the partition that is_______________call quickselect ( ; partition, then call quickselect ( , );once again, partition; at this point, so the pivot isthe 4th element, and thus the answer
Quickselect
86
example: find the median of 7 items ( )call quickselect ( ; partition; since , we want the
smallest element of , socall quickselect ( , ); partition; since we are inside the callquickselect ( , ), we want the 1st smallest element, so wecall quickselect ( , ), which immediately exits, returning10
Quickselect
87
quickselect complexityat each recursive step quickselect ignores one partition will thismake it faster than quicksort?in the worst case, quickselect behaves like quicksort, and hascomplexity
this occurs if the one partition is at each partitioning,and we have to look at all the terms in the other partition.
best case behavior is ________________occurs if each partition is ____________since quickselect ignores one partition at each step, itsruntime satisfies the recurrence
this leads to being linear
Quickselect
88
quickselect complexity (cont.)expected behavior
suppose we choose our pivot from theterms we are searchingsuppose lies between the 25th and 75th percentiles of theterms (i.e., is larger than 1/4 and smaller than 1/4 of theterms)this means that neither partition can contain more than 3/4 ofthe terms, so the partitions can't be too ;call such aon average, how many do we need to choose before we geta good one?
a randomly chosen is good with probability ½ - a goodpivot lies in the middle 50% of the termschoosing a good pivot is like tossing a coin and seeingheads
Quickselect
89
quickselect complexity (cont.)expected behavior (cont.)
the expected number of tosses to see a heads is twoto see this, let be the expected number of tossesbefore seeing a headstoss the coin; if it's heads, we're done; if it's tails (whichoccurs with probability 1/2) we have to toss it again, so
, whence
thus, on average, quickselect will take two ___________to reduce the array to at most 3/4 of the original size
Quickselect
90
quickselect complexity (cont.)expected behavior (cont.)
in terms of ,expected value of expected time toreduce the array
since each partitioning step requires work, and weexpect to need 2 of them to reduce the array size to
, we have
Quickselect
91
quickselect complexity (cont.)expected behavior (cont.)
consider the more general recurrence, where
at the level of the recursion, starting with ,there is a single problem of size at mostthe amount of work done at each level is thus at most
the recursion continues until
so , or
Quickselect
92
quickselect complexity (cont.)expected behavior (cont.)
thus, the total amount of work is bounded above by
thus, the best case and expected case behavior ofquickselect with a randomly chosen pivot is
Introsort
93
from D. R. Musser, Introspective sorting and selectionalgorithms, Software: Practice and Experience 27 (8) 983-993, 1997introsort is a of quicksort and heapsortintrosort starts with quicksort, but switches to heapsortwhenever the number of levels of recursion exceed aspecified threshold (e.g., ).if it switches to heapsort, the subarrays that remain to besorted will likely be much than the originalarraythis approach gives an complexity,while making the most of the efficiency of quicksort
Stability
94
a sorting algorithm is stable if it preserves the relative orderof keysstable sorts:
insertion sortmergesort
unstable sorts:selection sortShell sortquicksortheapsort
Stability
95
C++ Standard Library Sorts
96
sort(): introsort, guaranteed to bestable_sort(): mergesort, guaranteed to be stable and
time, if a -sized auxiliary array can be allocatedtime for an in-place sort, otherwise
partial_sort(): sort the largest (or smallest) items
String / Radix Sort
97
we can use the special structure of character strings todevise sorting algorithmshere, a character should be understood in a general sense,as an element of an alphabet, which is a collection of items(presumably all requiring the same number of fortheir representation)the alphabet is also assumed to have an ,so we may sort charactersas a special case, we can think of treating base- numbersas a string of digits from the range to ( -digit base-numbers)since the base of a number system is sometimes called the___________, the sorting algorithms we will discuss arefrequently called radix sorts
String Sorting: LSD
98
in least-significant digit (LSD) first sorting, we sort using thedigits (characters) from least- to most-significant:
String Sorting: LSD
99
algorithm:for d from least- to most-significant digit {
sort using counting sort on the least-significant digit}
observations:since counting sort is stable, LSD sorting is stable
the stability of counting sort is essential to making LSDsorting work
algorithm assumes all the strings have the same lengthtime complexity: there are passes in the outmost loop; ineach loop iteration, we apply counting sort to digits in therange to , requiring ; total work:space complexity: same as counting sort
String Sorting: LSD
100
why not go left-to-right?the same idea, but starting with the most significant digit,doesn't work:
why does right-to-left digit sorting work but left-to-right doesnot?
String Sorting: LSD
101
in the old days, we used Hollerith cards:
early versions of and COBOL had limits of 72characters per linethis left columns 73-80 free for the card numberthat way, if you dropped your deck of cards, you could go to...
String Sorting: LSD
102
IBM Model 082 card sorter
String Sorting: LSD
103
these machines used LSD:pile the disordered cards in the input hopper and sort by the__________ digit in the sequence numbernow take the sorted decks and stack them in orderplace the combined deck back in the input hopper and sortby the next-to-last digit in the sequence numberrepeat steps 2 and 3 until sorted
Sorting Algorithms Summary
104
Shell sort is subquadratic with a suitable increment sequenceShell's original increment sequence is, in fact, quadratic inthe worst case