demystifying coroutines and asynchronous · 2019-04-15 · the coroutine we write, delegates with...
TRANSCRIPT
![Page 1: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/1.jpg)
Demystifying Coroutines and Asynchronous Programming in Python
Mariano Anaya@rmarianoa
FOSDEM 2019 - Feb 03
![Page 2: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/2.jpg)
History● PEP-255: Simple generators● PEP-342: Coroutines via enhanced
generators● PEP-380: Syntax for delegating to a
sub-generator● PEP-492: Coroutines with async and
await syntax
![Page 3: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/3.jpg)
Generators
![Page 4: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/4.jpg)
![Page 5: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/5.jpg)
Generate elements, one at the time, and suspend...
● Save memory● Support iteration pattern, infinite
sequences, etc.
![Page 6: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/6.jpg)
Simple Generators● next() will advance until the next
yield○ Produce a value, & suspend
○ End? → StopIteration
![Page 7: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/7.jpg)
Coroutines
![Page 8: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/8.jpg)
![Page 9: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/9.jpg)
Can simple generators...
● ... suspend? ✔● … send/receive data from the
context?❌● … handle exceptions from the caller’s
context?❌
![Page 10: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/10.jpg)
Generators as coroutines
New methods!
<g>.send(<value>)
<g>.throw(<exception>)
<g>.close()
![Page 11: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/11.jpg)
Coroutines are syntactically like generators.
Syntactically equivalent, semantically different.
Coroutines via Enhanced Generators
![Page 12: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/12.jpg)
With .send(), the caller sends (receives) data to (from) the coroutine.
value = yield result
Coroutines via Enhanced Generators
![Page 13: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/13.jpg)
>>> c = coro()>>> next(c)>>> step = c.send(received)
def coro(): step = 0 while True: received = yield step step += 1 print(f"Received: {received}")
![Page 14: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/14.jpg)
>>> c = coro()>>> next(c) # important!0
def coro(): step = 0 while True: received = yield step step += 1 print(f"Received: {received}")
![Page 15: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/15.jpg)
>>> step = c.send(100)
def coro(): step = 0 while True: received = yield step step += 1 print(f"Received: {received}")
![Page 16: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/16.jpg)
>>> step = c.send(100)Received: 100
def coro(): step = 0 while True: received = yield step step += 1 print(f"Received: {received}")
![Page 17: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/17.jpg)
>>> step = c.send(100)Received: 100>>> step1
def coro(): step = 0 while True: received = yield step step += 1 print(f"Received: {received}")
![Page 18: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/18.jpg)
>>> c.throw(ValueError)---------------------ValueError Traceback (most recent call last)----> 1 step = c.throw(ValueError) 5 step = 0 6 while True:----> 7 received = yield step 8 step += 1 9 print(f"Received: {received}")
![Page 19: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/19.jpg)
Can we do better?
![Page 20: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/20.jpg)
Better Coroutines
![Page 21: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/21.jpg)
![Page 22: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/22.jpg)
Delegating to a Sub-Generator● Enhancements○ Generators can now return
values!○ yield from
![Page 23: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/23.jpg)
Generators - Return values→ StopIteration.value
>>> def gen():...: yield 1...: yield 2...: return 42
>>> g = gen()>>> next(g)1>>> next(g)2>>> next(g)--------------------------StopIteration Traceback (most recent call last)StopIteration: 42
![Page 24: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/24.jpg)
yield from - Basic
Something in the form
yield from <iterable>
Can be thought of as
for e in <iterable>: yield e
![Page 25: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/25.jpg)
yield from - More
● Nested coroutines: .send(), and .throw() are passed along.
● Capture return values
value = yield from coroutine(...)
![Page 26: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/26.jpg)
yield from Example
![Page 27: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/27.jpg)
def internal(name, start, end): for i in range(start, end): value = yield i print(f"{name} got: {value}") print(f"{name} finished at {i}") return end
def general(): start = yield from internal("first", 1, 5) end = yield from internal("second", start, 10) return end
![Page 28: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/28.jpg)
>>> g = general()>>> next(g)
![Page 29: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/29.jpg)
>>> g = general()>>> next(g)1
![Page 30: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/30.jpg)
>>> g = general()>>> next(g)1
>>> g.send("1st value sent to main coroutine")
![Page 31: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/31.jpg)
>>> g = general()>>> next(g)1
>>> g.send("1st value sent to main coroutine")first got: 1st value sent to main coroutine2
![Page 32: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/32.jpg)
...
>>> next(g)first got: Nonefirst finished at 45
![Page 33: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/33.jpg)
...
>>> g.send("value sent to main coroutine")second got: value sent to main coroutine6
![Page 34: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/34.jpg)
yield from - Recap● Better way of combining
generators/coroutines.● Enables chaining generators and
many iterables together.
![Page 35: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/35.jpg)
Issues & limitations
![Page 36: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/36.jpg)
async def / await
![Page 37: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/37.jpg)
![Page 38: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/38.jpg)
yield from → await# py [email protected] coroutine(): yield from asyncio.sleep(1)
# py 3.5+async def coroutine(): await asyncio.sleep(1)
![Page 39: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/39.jpg)
await~ yield from, except that:
● Doesn’t accept generators that aren’t coroutines.
● Accepts awaitable objects○ __await__()
![Page 40: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/40.jpg)
asyncio● Event loop → scheduled & run coroutines
○ Update them with send()/throw().● The coroutine we write, delegates with
await, to some other 3rd party generator, that will do the actual I/O.
● Calling await gives the control back to the scheduler.
![Page 41: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/41.jpg)
Summary● Coroutines evolved from generators, but
they’re conceptually different ideas.● yield from → await: more powerful
coroutines (&types).● A chain of await calls ends with a yield.
![Page 42: Demystifying Coroutines and Asynchronous · 2019-04-15 · The coroutine we write, delegates with await, to some other 3rd party generator, that will do the actual I/O. Calling await](https://reader034.vdocuments.us/reader034/viewer/2022050103/5f42633170c5e52f1b7d0072/html5/thumbnails/42.jpg)
Thank You!Mariano Anaya
@rmarianoa