Download - Parallel programming patterns (UA)
![Page 2: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/2.jpg)
Зміст
- Тренд- Основні терміни- Managing state- Паралелізм- Засоби
![Page 3: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/3.jpg)
Вчора
![Page 4: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/4.jpg)
Сьогодні
![Page 5: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/5.jpg)
Завтра
![Page 6: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/6.jpg)
Що відбувається?
- Ріст частоти CPU вповільнився- Через фізичні обмеження- Free lunch is over- ПЗ більше не стає швидшим саме по собі
![Page 7: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/7.jpg)
Сучасні тренди
- Manycore, multicore- GPGPU, GPU acceleration, heterogeneous
computing- Distributed computing, HPC
![Page 8: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/8.jpg)
Основні поняття
- Concurrency- Many interleaved threads of control
- Parallelism- Same result, but faster
- Concurrency != Parallelism- It is not always necessary to care about
concurrency while implementing parallelism- Multithreading- Asynchrony
![Page 9: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/9.jpg)
Задачі
- CPU-bound- number crunching
- I/O-bound- network, disk
![Page 10: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/10.jpg)
Стан
- Shared- accessible by more than one thread- sharing is transitive
- Private- used by single thread only
![Page 11: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/11.jpg)
Task-based program
Application
Tasks (CPU, I/O)
Runtime (queuing, scheduling)
Processors (threads, processes)
![Page 12: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/12.jpg)
Managing state
![Page 13: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/13.jpg)
Isolation
- Avoiding shared state- Own copy of state- Examples:
- process isolation- intraprocess isolation- by convention
![Page 14: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/14.jpg)
Immutability
- Multiple read -- not a problem!- All functions are pure- Requires immutable collections- Functional way: Haskell, F#, Lisp
![Page 15: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/15.jpg)
Synchronization
- The only thing that remains to deal with shared mutable state
- Kinds:- data synchronization- control synchronization
![Page 16: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/16.jpg)
Data synchronization
- Why? To avoid race conditions and data corruption
- How? Mutual exclusion- Data remains consistent- Critical regions
- locks, monitors, critical sections, spin locks- Code-centered
- rather than associated with data
![Page 17: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/17.jpg)
Critical region|Thread 1|// ...|lock (locker)|{| // ...| data.Operation();| // ...|}|// ...|||
|Thread 2|// ...|||||||lock (locker)|{| // ...| data.Operation();| // ...|}
![Page 18: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/18.jpg)
Control synchronization
- To coordinate control flow- exchange data- orchestrate threads
- Waiting, notifications- spin waiting- events- alternative: continuations
![Page 19: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/19.jpg)
Three ways to manage state
- Isolation: simple, loosely coupled, highly scalable, right data structures, locality- Immutability: avoids sync- Synchronization: complex, runtime overheads, contention
- in that order
![Page 20: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/20.jpg)
Паралелізм
![Page 21: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/21.jpg)
Підходи до розбиття задач
- Data parallelism- Task parallelism- Message based parallelism
![Page 22: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/22.jpg)
Data parallelism
How?
- Data is divided up among hardware processors- Same operation is performed on elements - Optionally -- final aggregation
![Page 23: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/23.jpg)
Data parallelism
When?
- Large amounts of data- Processing operation is costly- or both
![Page 24: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/24.jpg)
Data parallelism
Why?
- To achieve speedup
- For example, with GPU acceleration:- hours instead of days!
![Page 25: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/25.jpg)
Data parallelism
Embarrassingly parallel problems- parallelizable loops- image processing
Non-embarrassingly parallel problems- parallel QuickSort
![Page 26: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/26.jpg)
Data parallelism
...
Thread 1
...
Thread 2
![Page 27: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/27.jpg)
Data parallelism
Structured parallelism
- Well defined begin and end points- Examples:
- CoBegin- ForAll
![Page 28: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/28.jpg)
CoBegin
var firstDataset = new DataItem[1000];var secondDataset = new DataItem[1000];var thirdDataset = new DataItem[1000];
Parallel.Invoke( () => Process(firstDataset), () => Process(secondDataset), () => Process(thirdDataset) );
![Page 29: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/29.jpg)
Parallel For
var items = new DataItem[1000 * 1000];// ...Parallel.For(0, items.Length, i => { Process(items[i]); });
![Page 30: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/30.jpg)
Parallel ForEach
var tickers = GetNasdaqTickersStream();Parallel.ForEach(tickers, ticker => { Process(ticker); });
![Page 31: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/31.jpg)
Striped Partitioning
Thread 1 Thread 2
...
![Page 32: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/32.jpg)
Iterate complex data structures
var tree = new TreeNode();// ...Parallel.ForEach( TraversePreOrder(tree), node => { Process(node); });
![Page 33: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/33.jpg)
Iterate complex data
...
Thread 1
Thread 2
![Page 34: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/34.jpg)
Declarative parallelismvar items = new DataItem[1000 * 1000];// ...var validItems = from item in items.AsParallel() let processedItem = Process(item) where processedItem.Property > 42 select Convert(processedItem);
foreach (var item in validItems){ // ...}
![Page 35: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/35.jpg)
Data parallelism
Challenges
- Partitioning- Scheduling- Ordering- Merging- Aggregation- Concurrency hazards: data races, contention
![Page 36: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/36.jpg)
Task parallelism
How?
- Programs are already functionally partitioned: statements, methods etc.- Run independent pieces in parallel- Control synchronization- State isolation
![Page 37: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/37.jpg)
Task parallelism
Why?
- To achieve speedup
![Page 38: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/38.jpg)
Task parallelism
Kinds- Structured
- clear begin and end points- Unstructured
- often demands explicit synchronization
![Page 39: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/39.jpg)
Fork/join
- Fork: launch tasks asynchronously- Join: wait until they complete- CoBegin, ForAll- Recursive decomposition
![Page 40: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/40.jpg)
Fork/join
Task 1
Task 2
Task 3
SeqSeq
![Page 41: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/41.jpg)
Fork/join
Parallel.Invoke( () => LoadDataFromFile(), () => SavePreviousDataToDB(), () => RenewOtherDataFromWebService());
![Page 42: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/42.jpg)
Fork/join
Task loadData = Task.Factory.StartNew(() => { // ... });Task saveAnotherDataToDB = Task.Factory.StartNew(() => { // ... });// ...Task.WaitAll(loadData, saveAnotherDataToDB);// ...
![Page 43: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/43.jpg)
Fork/join
void Walk(TreeNode node) { var tasks = new[] { Task.Factory.StartNew(() => Process(node.Value)), Task.Factory.StartNew(() => Walk(node.Left)), Task.Factory.StartNew(() => Walk(node.Right)) }; Task.WaitAll(tasks);}
![Page 44: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/44.jpg)
Fork/join recursive
Root
SeqSeq Left
Right
Node
Left
Right
Node
Left
Right
![Page 45: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/45.jpg)
Dataflow parallelism: Futures
Task<DataItem[]> loadDataFuture = Task.Factory.StartNew(() => { //... return LoadDataFromFile(); });
var dataIdentifier = SavePreviousDataToDB();RenewOtherDataFromWebService(dataIdentifier);//...DisplayDataToUser(loadDataFuture.Result);
![Page 46: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/46.jpg)
Dataflow parallelism: Futures
Seq
Future
Seq Seq
![Page 47: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/47.jpg)
Dataflow parallelism: Futures
Seq
Future
Seq
Future
Seq
Future
Seq Seq
![Page 48: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/48.jpg)
Continuations
Seq
Task
Seq Seq
Task
Task
![Page 49: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/49.jpg)
Continuationsvar loadData = Task.Factory.StartNew(() => { return LoadDataFromFile(); });
var writeToDB = loadData.ContinueWith(dataItems => { WriteToDatabase(dataItems.Result); });
var reportToUser = writeToDB.ContinueWith(t => { // ... });reportProgressToUser.Wait();
![Page 50: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/50.jpg)
Producer/consumerpipeline
lines parsedlines DB
reading parsing storing
![Page 51: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/51.jpg)
Producer/consumerpipeline
lines
parsedlines
DB
![Page 52: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/52.jpg)
Producer/consumer
var lines = new BlockingCollection<string>();
Task.Factory.StartNew(() => { foreach (var line in File.ReadLines(...)) lines.Add(line); lines.CompleteAdding(); });
![Page 53: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/53.jpg)
Producer/consumer
var dataItems = new BlockingCollection<DataItem>();
Task.Factory.StartNew(() => { foreach (var line in lines.GetConsumingEnumerable() ) dataItems.Add(Parse(line)); dataItems.CompleteAdding(); });
![Page 54: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/54.jpg)
Producer/consumer
var dbTask = Task.Factory.StartNew(() => { foreach (var item in dataItems.GetConsumingEnumerable() ) WriteToDatabase(item); });
dbTask.Wait();
![Page 55: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/55.jpg)
Task parallelism
Challenges
- Scheduling- Cancellation- Exception handling- Concurrency hazards: deadlocks, livelocks, priority inversions etc.
![Page 56: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/56.jpg)
Message based parallelism
- Accessing shared state vs. local state- No distinction, unfortunately- Idea: encapsulate shared state changes into
messages- Async events- Actors, agents
![Page 57: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/57.jpg)
Засоби
![Page 58: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/58.jpg)
Concurrent data structures
- Concurrent Queues, Stacks, Sets, Lists- Blocking collections, - Work stealing queues- Lock free data structures- Immutable data structures
![Page 59: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/59.jpg)
Synchronization primitives
- Critical sections, - Monitors, - Auto- and Manual-Reset Events,- Coundown Events, - Mutexes, - Semaphores, - Timers, - RW locks- Barriers
![Page 60: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/60.jpg)
Thread local state
- A way to achieve isolation
var parser = new ThreadLocal<Parser>( () => CreateParser());
Parallel.ForEach(items, item => parser.Value.Parse(item));
![Page 61: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/61.jpg)
Thread pools
ThreadPool.QueueUserWorkItem(_ => { // do some work });
![Page 62: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/62.jpg)
AsyncTask.Factory.StartNew(() => { //... return LoadDataFromFile(); }) .ContinueWith(dataItems => { WriteToDatabase(dataItems.Result); }) .ContinueWith(t => { // ... });
![Page 63: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/63.jpg)
Asyncvar dataItems = await LoadDataFromFileAsync();
textBox.Text = dataItems.Count.ToString();
await WriteToDatabaseAsync(dataItems);
// continue work
![Page 64: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/64.jpg)
Технології
- TPL, PLINQ, C# async, TPL Dataflow- PPL, Intel TBB, OpenMP- CUDA, OpenCL, C++ AMP- Actors, STM- Many others
![Page 65: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/65.jpg)
Підсумок
- Програмування для багатьох CPU- Concurrency != parallelism- CPU-bound vs. I/O-bound tasks- Private vs. shared state
![Page 66: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/66.jpg)
Підсумок
- Managing state:- Isolation - Immutability- Synchronization
- Data: mutual exclusion- Control: notifications
![Page 67: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/67.jpg)
Підсумок
- Паралелізм:- Data parallelism: scalable- Task parallelism: less scalable- Message based parallelism
![Page 68: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/68.jpg)
Підсумок
- Data parallelism- CoBegin- Parallel ForAll- Parallel ForEach- Parallel ForEach over complex data structures- Declarative data parallelism
- Challenges: partitioning, scheduling, ordering, merging, aggregation, concurrency hazards
![Page 69: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/69.jpg)
Підсумок
- Task parallelism: structured, unstructured- Fork/Join
- CoBegin- Recursive decomposition
- Futures- Continuations- Producer/consumer (pipelines)
- Challenges: scheduling, cancellation, exceptions, concurrency hazards
![Page 70: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/70.jpg)
Підсумок
- Засоби/інструменти- Компілятори, бібліотеки- Concurrent data structures- Synchronization primitives- Thread local state- Thread pools- Async invocations- ...
![Page 71: Parallel programming patterns (UA)](https://reader036.vdocuments.us/reader036/viewer/2022062307/5585af93d8b42a711a8b5276/html5/thumbnails/71.jpg)
Q/A