external sorting
DESCRIPTION
External Sorting. Outline. In this topic we will look at external sorting: Suppose we are sorting entries stored in secondary memory What if we can’t load everything into main memory? Can we sort sections of the unsorted data?. Situation. 8.10. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/1.jpg)
ECE 250 Algorithms and Data Structures
Douglas Wilhelm Harder, M.Math. LELDepartment of Electrical and Computer EngineeringUniversity of WaterlooWaterloo, Ontario, Canada
© 2006-2013 by Douglas Wilhelm Harder. Some rights reserved.
External Sorting
![Page 2: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/2.jpg)
2External Sorting
Outline
In this topic we will look at external sorting:– Suppose we are sorting entries stored in secondary memory– What if we can’t load everything into main memory?– Can we sort sections of the unsorted data?
![Page 3: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/3.jpg)
3External Sorting
Situation
Suppose we are sorting n entries stored in secondary memory (a hard drive or tape drive) than cannot be loaded into main memory– These secondary storages are block addressable– Assume each block is 4 KiB and there are b entries per block
– Note, 4 KiB blocks are a common division of both hard drives and of main memory pages
8.10
![Page 4: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/4.jpg)
4External Sorting
Naming Convention
Convention:– Lower-case variables will sort the number of entries being sorted
• Entries being sorted, entries per block, etc.– Upper-case variables will store the size of larger structures
8.10.1
![Page 5: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/5.jpg)
5External Sorting
Strategy
First, assume we can load M blocks at a time into main memory– Divide the data we are sorting into N sections of M blocks– Each section has m = Mb entries and n = NMb– Here, we are loading the first M = 10 blocks into main memory
8.10.2
![Page 6: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/6.jpg)
6External Sorting
Strategy
On these m = Mb entries, apply an in-place sorting algorithm such as quicksort or merge sort– The run time is Q( m ln(m) )
8.10.2
![Page 7: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/7.jpg)
7External Sorting
Strategy
Write these M sorted blocks back into secondary memory
8.10.2
![Page 8: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/8.jpg)
8External Sorting
Strategy
Repeat this process for all N blocks– The run time is Q( Nm ln(m) ) = Q( n ln(m) )
8.10.2
![Page 9: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/9.jpg)
9External Sorting
Strategy
Next, load the first block of each of the N sections into corresponding input buffers allocated in main memory– If we had 2 GiB of main memory, we could load 512 such input buffers– We would need to sort more than 1 EiB before this was not possible
– The math:
8.10.2
3119
12
2GiB 22GiB 2GiB 2GiB 2 1EiB4KiB 2
![Page 10: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/10.jpg)
10External Sorting
Strategy
Now, perform an N-way merge of these N sorted blocks into a single block-sized output buffer
8.10.2
![Page 11: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/11.jpg)
11External Sorting
Strategy
When this output buffer is full, we will write it into secondary memory and start again with an empty buffer– Because we are using merging, we cannot do this in-place on
secondary memory—we will require Q(n) additional secondary memory
8.10.2
![Page 12: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/12.jpg)
12External Sorting
Strategy
Whenever one of the N input buffers is entirely merged, we load the next block from the corresponding section into the input buffer
8.10.2
![Page 13: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/13.jpg)
13External Sorting
Strategy
We continue this process of storing the merged block into secondary memory and reading subsequent blocks into main memory
8.10.2
![Page 14: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/14.jpg)
14External Sorting
Strategy
We continue this process of storing the merged block into secondary memory and reading subsequent blocks into main memory– At some point we will be merging the last blocks of each of the sections
8.10.2
![Page 15: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/15.jpg)
15External Sorting
Strategy
When the last block is written out, the collection of MN blocks now contain a sorted list of the initial n entries
8.10.2
![Page 16: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/16.jpg)
16External Sorting
Run-time Analysis
With merge sort, we saw that merging was an Q(n1 + n2) operation when merging lists of size n1 and n2 – From slide 8, this suggests the run time is Q( n ln(m) ) – If m is sufficiently small, this suggests the run-time is Q(n) !!!
8.10.3
![Page 17: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/17.jpg)
17External Sorting
Run-time Analysis
This, however, is a mathematical fallacy…– A binary merge can be performed in linear time, an N-way merge cannot– The best we can hope for is using a binary min-heap
• Each will require O(ln(N)) time for a total of Q( n ln(N) )
Thus, the total run time is Q( n ln(m) + n ln(N) ) = Q( n ln(mN) ) = Q( n ln(n) )
8.10.3
Recall that n = mN
![Page 18: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/18.jpg)
18External Sorting
Run-time Analysis
To be complete, we should also consider the run time associated with loading and saving data to the disk:– Each item of data is copied from or to the external disk four times– We will assume the memory size of each item is fixed and reasonable– Thus, the run time of these operations is Q(n)– If we are considering the size of the objects to be variable, say k bytes,
we would have to revise our run time to Q( n ln(n) + nk )
8.10.3
![Page 19: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/19.jpg)
19External Sorting
Additional Remarks
With 4 KiB blocks and 2 GiB of available main memory, we determined we could sort up to 1 EiB of data
What if our available memory was smaller, say 1 MiB?
Solution?– Divide the data into 256 MiB blocks– Sort each of these individually use external sorting– Now merge these using the same strategy
• We are restricted by the number of blocks we can merge– This process can be repeated arbitrarily often—doing so K times allows
us to sort
208
12
1MiB 21MiB 1MiB 1MiB 2 256MiB4KiB 2
208
12
1MiB 21MiB 1MiB 2 MiB4KiB 2
KKK
8.10.4
![Page 20: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/20.jpg)
20External Sorting
Additional Remarks
We have focused on one implementation of using block-sized buffers—optimizations, however, are always possible:– Performing the initial sorting (quick sort) in parallel– Storing at least a portion of the data in solid-state drives– Using larger input and output buffers (useful if the data is contiguous on
the hard drive)
8.10.4
![Page 21: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/21.jpg)
21External Sorting
Summary
This topic covered external sorting– Divide the n entries being sorted into N sections that fit in main memory– Sort each of the N section and write to secondary memory– Load blocks from each section as necessary into main memory– Merge the sections using an N-way merge– Write the merged blocks to secondary memory– The run time is Q(n ln(n))
![Page 22: External Sorting](https://reader036.vdocuments.us/reader036/viewer/2022070422/56816552550346895dd7cc75/html5/thumbnails/22.jpg)
22External Sorting
References
Donald E. Knuth, The Art of Computer Programming, Volume 3: Sorting and Searching, 2nd Ed., Addison Wesley, 1998, §5.4, pp.248-379.
Wikipedia, https://en.wikipedia.org/wiki/External_sorting
Special thanks to Prof. Ran Ginosar who made some observations and suggestions and who led me to finding an error in my calculations
These slides are provided for the ECE 250 Algorithms and Data Structures course. The material in it reflects Douglas W. Harder’s best judgment in light of the information available to him at the time of preparation. Any reliance on these course slides by any party for any other purpose are the responsibility of such parties. Douglas W. Harder accepts no responsibility for damages, if any, suffered by any party as a result of decisions made or actions based on these course slides for any other purpose than that for which it was intended.