concurrency in python · concurrency approaches multiprocessing concurrency of os processes module...
TRANSCRIPT
![Page 1: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/1.jpg)
Concurrency in PythonConcepts, frameworks and best practices
PyCon DE
Stefan Schwarzer, [email protected]
Karlsruhe, Germany, 2018-10-26
![Page 2: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/2.jpg)
About me
Using Python since 1999
Software developer since 2000
Freelancer since 2005
Book “Workshop Python”, Addison-Wesley,using the then brand new Python 2.2 ;-)
About 15 conference talks
Maintainer of ftputil (high-level FTP client library)since 2002
Concurrency in Python 2 / 47
![Page 3: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/3.jpg)
Overview
Basics
Concurrency approaches
Race conditions
Deadlocks
Queues
Higher-level concurrency approaches
Best practices
Concurrency in Python 3 / 47
![Page 4: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/4.jpg)
Basics
reasons, terms
![Page 5: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/5.jpg)
Reasons for concurrency
CPU intensive tasksSpeed up algorithms by executing parts in parallel.
Input/outputOther parts of the program can run while waiting for I/O.
ReactivityWhile a GUI application executes some lengthy operation,the application should still accept user interaction.
Concurrency in Python 5 / 47
![Page 6: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/6.jpg)
Terms
ResourceAnything that’s used by an execution thread (not necessarilyan OS thread), for example simple variables, data structures,files or network sockets.
ConcurrencyThere are multiple execution threads. They don’t have toprogress at the same time.
ParallelismExecution threads run at the very same time (for exampleon different CPU cores).
Atomic operationA task that can’t be interrupted by another execution thread
Concurrency in Python 6 / 47
![Page 7: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/7.jpg)
Concurrency approaches
multithreading, multiprocessing, event loop
![Page 8: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/8.jpg)
Concurrency approachesMultithreading
Concurrency of OS threads in a single process
Module threading in the standard library
Threads can share data in process memory
For CPython the global interpreter lock (GIL) applies
The GIL prevents the parallel execution of Python code.The GIL is released during I/O operations.Also, C extensions can release the GIL.
Concurrency in Python 8 / 47
![Page 9: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/9.jpg)
Concurrency approachesMultiprocessing
Concurrency of OS processes
Module multiprocessing in the standard library
Data transfer between processes via messagesor shared memory
When transferring messages, they must be serialized.This is additional work.
Advantage of multiprocessing: no limitation of parallelexecution, not even for CPU-limited work. The GILis per Python process.
Concurrency in Python 9 / 47
![Page 10: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/10.jpg)
Concurrency approachesEvent loop
Loop (“main loop”) detects events (examples: mouse click,incoming network data)
Variants:
Depending on the event, a “handler” is called and processesthe event. Control returns to the main loop after the handlerexecution.Code looks sequential, but execution is switched to other codeif the event loop has to wait for I/O.Both variants may be used in the same program.
An event loop implementation is in the package asyncio
in the standard library.
Concurrency in Python 10 / 47
![Page 11: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/11.jpg)
Race conditions
definition, code example, explanation, fix
![Page 12: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/12.jpg)
Race conditionsDefinition
While a resource is modified by an execution thread,another execution thread modifies or reads the resource.
Concurrency in Python 12 / 47
![Page 13: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/13.jpg)
Race conditionsCode without protection against concurrent access
import threading, time # ‘sys.setswitchinterval‘ omitted
counter = 0
def count():
global counter
for _ in range(100):
counter += 1
threads = []
for _ in range(100):
thread = threading.Thread(target=count)
thread.start() # Start thread. Don’t confuse with ‘run‘.
threads.append(thread)
for thread in threads:
thread.join() # Wait until thread is finished.
print("Total:", counter)
Concurrency in Python 13 / 47
![Page 14: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/14.jpg)
Race conditionsOutput without protection against concurrent access
$ python3 race_condition.py
Total: 9857
$ python3 race_condition.py
Total: 9917
$ python3 race_condition.py
Total: 9853
$ python3 race_condition.py
Total: 9785
$ python3 race_condition.py
Total: 9972
$ python3 race_condition.py
Total: 9731
Concurrency in Python 14 / 47
![Page 15: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/15.jpg)
Race conditionsExplanation – race condition because of concurrent access
This is only one of many possibilities.
Thread 2 reads the earlier value of counter because thread 1hasn’t stored the new value yet.
Concurrency in Python 15 / 47
![Page 16: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/16.jpg)
Race conditionsCode with protection against concurrent access
import threading, time # ‘sys.setswitchinterval‘ omitted
counter = 0
lock = threading.Lock()
def count_with_lock():
global counter
for _ in range(100):
with lock:
counter += 1 # Atomic operation
threads = []
for _ in range(100):
thread = threading.Thread(target=count_with_lock)
thread.start()
threads.append(thread)
...Concurrency in Python 16 / 47
![Page 17: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/17.jpg)
Deadlocks
definition, code example
![Page 18: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/18.jpg)
DeadlocksDefinition
A deadlock happens if execution threads mutuallyclaim resources that the other execution threads need.
Example:
Both thread 1 and 2 need resources A and B to finish a task.
Thread 1 already holds resource A and wants resource B.
Thread 2 already holds resource B and wants resource A.
→ Deadlock!
Concurrency in Python 18 / 47
![Page 19: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/19.jpg)
DeadlocksExample code
# Thread 1
with input_lock: # 1st
with output_lock: # blocks
input_line = input_fobj.readline()
# Process input ...
output_fobj.write(output_line)
# Thread 2
with output_lock: # 2nd
with input_lock: # blocks
input_line = input_fobj.readline()
# Process input ...
output_fobj.write(output_line)
Concurrency in Python 19 / 47
![Page 20: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/20.jpg)
Queues
code example with worker threads
![Page 21: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/21.jpg)
QueuesSchema for the following example
Principle: put and get are atomic operations.
Concurrency in Python 21 / 47
![Page 22: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/22.jpg)
QueuesSetup
import logging, queue, random, threading, time
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("queue_example")
WORKER_COUNT = 10
JOB_COUNT = 100
# Needed to shut down threads without race conditions.
STOP_TOKEN = object()
job_queue = queue.Queue()
class Job:
def __init__(self, number):
self.number = number
Concurrency in Python 22 / 47
![Page 23: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/23.jpg)
QueuesWorker thread
class Worker(threading.Thread):
def run(self):
while True:
job = job_queue.get(block=True)
if job is STOP_TOKEN:
break
self._process_job(job)
def _process_job(self, job):
# Wait between 0 and 0.01 seconds.
time.sleep(random.random() / 100.0)
# Atomic output
logger.info("Job number {:d}".format(job.number))
Concurrency in Python 23 / 47
![Page 24: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/24.jpg)
QueuesCreation and execution of jobs
def main():
workers = []
# Create and start workers.
for _ in range(WORKER_COUNT):
worker = Worker()
worker.start()
workers.append(worker)
# Schedule jobs for workers.
for i in range(JOB_COUNT):
job_queue.put(Job(i))
# Schedule stopping of workers.
for _ in range(WORKER_COUNT):
job_queue.put(STOP_TOKEN)
# Wait for workers to finish.
for worker in workers:
worker.join()
Concurrency in Python 24 / 47
![Page 25: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/25.jpg)
Higher-level concurrency approaches
concurrent.futures, active objects, process networks
![Page 26: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/26.jpg)
concurrent.futuresExample
import concurrent.futures
import logging
import random
import time
WORKER_COUNT = 10
JOB_COUNT = 100
class Job:
def __init__(self, number):
self.number = number
Concurrency in Python 26 / 47
![Page 27: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/27.jpg)
concurrent.futuresExample
def process_job(job):
# Wait between 0 and 0.01 seconds.
time.sleep(random.random() / 100.0)
# Atomic output
logger.info("Job number {:d}".format(job.number))
def main():
with concurrent.futures.ThreadPoolExecutor(
max_workers=WORKER_COUNT) as executor:
# Distribute jobs.
futures = [executor.submit(process_job, Job(i))
for i in range(JOB_COUNT)]
# Wait for work to finish.
for future in concurrent.futures.as_completed(futures):
pass
Concurrency in Python 27 / 47
![Page 28: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/28.jpg)
concurrent.futuresComparison with queue example
process job is now a function, no need to inherit fromthreading.Thread and implement run
No queue needed
No error-prone token handling needed to stop the workersat the right time
→ Use concurrent.futures if you can! :-)
Concurrency in Python 28 / 47
![Page 29: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/29.jpg)
Active objects
Principle: Locks, queues or other synchronization mechanismsare not part of the API of an object.
Synchronization, as far as needed, is hidden in high-levelmethods.
Concurrency in Python 29 / 47
![Page 30: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/30.jpg)
Active objectsExample – constructor
import queue
import threading
STOP_TOKEN = object()
class Adder:
def __init__(self):
self._in_queue = queue.Queue()
self._out_queue = queue.Queue()
self._worker_thread = threading.Thread(
target=self._work)
self._worker_thread.start()
Concurrency in Python 30 / 47
![Page 31: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/31.jpg)
Active objectsExample – internal method
def _work(self):
while True:
work_item = self._in_queue.get(block=True)
if work_item is STOP_TOKEN:
break
result = work_item + 1000
self._out_queue.put(result)
Concurrency in Python 31 / 47
![Page 32: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/32.jpg)
Active objectsExample – public methods
def submit(self, work_item):
self._in_queue.put(work_item)
def next_result(self):
return self._out_queue.get(block=True)
def stop(self):
self._in_queue.put(STOP_TOKEN)
self._worker_thread.join()
Concurrency in Python 32 / 47
![Page 33: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/33.jpg)
Active objectsExample – usage
def main():
ITEM_COUNT = 100
adder = Adder()
for i in range(ITEM_COUNT):
# Doesn’t block
adder.submit(i)
# Do other things.
# ...
# Collect results.
for _ in range(ITEM_COUNT):
# May block
print(adder.next_result())
# May block
adder.stop()
Concurrency in Python 33 / 47
![Page 34: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/34.jpg)
Process networks
Processes receive input data and/or send output data.
Data transfer between processes by message passing
Processes can use different programming languagesif they use a message format that the communicatingprocesses understand.
Some overhead due to data serialization and protocols
Concurrency in Python 34 / 47
![Page 35: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/35.jpg)
Process networksWith broker
Processes communicate with a broker service, but notwith each other.
Broker
Process Process Process Process
Broker protocol examples: AMQP, MQTT
Declarative configuration
Message persistence (optional)
Concurrency in Python 35 / 47
![Page 36: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/36.jpg)
Process networksWithout broker
Processes communicate directly.
Process Process
Process
Process
Example: ZeroMQ
Concurrency in Python 36 / 47
![Page 37: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/37.jpg)
Best practices
caveats, general design advice, approaches, shared state
![Page 38: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/38.jpg)
Best practicesCaveats
The following “best practices” aren’t necessarily written downin books or online, but are my recommendations.
Different advice may apply to different areas of your code.
Concurrency in Python 38 / 47
![Page 39: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/39.jpg)
Best practicesGeneral design advice
Concurrency is an optimization.Like other optimizations, use it only if necessary.
Try to keep code simple and easy to understand.In many cases this would mean queues or higher-level APIsto communicate between threads or processes.
If you use low-level APIs, hide them. Don’t make locks,queues etc. a part of the public interface.
Concurrency in Python 39 / 47
![Page 40: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/40.jpg)
Best practicesChoose a concurrency approach
I/O-limited concurrencymultithreadingasyncio (for many concurrent tasks)process networks
CPU-limited concurrencymultiprocessingmultithreading (if using extensions that can release the GIL)process networks
GUI frameworksusually come with their own event loop
Concurrent processes in different languagesprocess networks
Concurrency in Python 40 / 47
![Page 41: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/41.jpg)
Best practicesShared state
Be extremely careful not to read shared state while it maybe written. Even query methods may be problematic if theyimplicitly update an internal cache of an object, for example.
Make sure the APIs you use from multiple threads arethread-safe. You can only count on the documentationbecause the code may be different in the next version.
Try to avoid shared state. Pass immutable objects or set upthe state before starting threads that access the state.
Concurrency involving shared state is difficult to test.Don’t assume your code doesn’t have concurrency issues onlybecause it seems to run fine. Invest some time to createa solid design. Have your code reviewed.
Concurrency in Python 41 / 47
![Page 42: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/42.jpg)
Thank you for your attention! :-)
Questions?
Remarks?
Discussion?
https://sschwarzer.com
Concurrency in Python 42 / 47
![Page 43: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/43.jpg)
Appendices
links, asyncio example
![Page 44: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/44.jpg)
Links
Dr. Dobb’s Parallel Computinghttp://www.drdobbs.com/parallel (overview page)http://www.drdobbs.com/212903586 (introduction)
“The problem with threads”https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
Design recommendationshttps://stackoverflow.com/questions/1190206/, especiallyhttps://stackoverflow.com/questions/1190206/threading-in-python/1192114#1192114
Active object patternhttp://www.drdobbs.com/225700095
“Notes on structured concurrency”https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful
Concurrency in Python 44 / 47
![Page 45: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/45.jpg)
asyncioExample – Setup
import asyncio
import logging
import random
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("asyncio_example")
JOB_COUNT = 100
class Job:
def __init__(self, number):
self.number = number
Concurrency in Python 45 / 47
![Page 46: Concurrency in Python · Concurrency approaches Multiprocessing Concurrency of OS processes Module multiprocessing in the standard library Data transfer between processes via messages](https://reader035.vdocuments.us/reader035/viewer/2022071005/5fc2dc95037b8b36b45f1133/html5/thumbnails/46.jpg)
asyncioExample – asynchronous code
async def process_job(job):
# Wait between 0 and 0.01 seconds.
await asyncio.sleep(random.random() / 100.0)
logger.info("Job number {:d}".format(job.number))
def main():
loop = asyncio.get_event_loop()
tasks = []
for i in range(JOB_COUNT):
task = loop.create_task(process_job(Job(i)))
tasks.append(task)
for task in tasks:
# Similar to ‘Thread.start‘ plus ‘Thread.join‘
loop.run_until_complete(task)
loop.close()
Concurrency in Python 46 / 47