Download - Lesson learnt from WebP. What’s next?
![Page 2: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/2.jpg)
Plan
● lessons learnt from VP8 -> WebP codec
● research direction and experiments for “WebP v2”
● results (+demo?)
![Page 3: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/3.jpg)
Motivation
WebP, HEIF, AVIF ...
![Page 4: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/4.jpg)
Motivation
WebP, HEIF, AVIF …
most recent Image codecs originate from Video codec.
![Page 5: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/5.jpg)
Motivation
WebP, HEIF, AVIF …
most recent Image codecs originate from Video codec.
Is it a always a good choice?
![Page 6: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/6.jpg)
Lessons learnt from VP8 -> WebP
![Page 7: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/7.jpg)
Lessons learnt from VP8 -> WebP
Two main use-cases for image compression:
● “Capture” [device -> storage / CDN]
![Page 8: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/8.jpg)
Lessons learnt from VP8 -> WebP
Two main use-cases for image compression:
● “Capture” [device -> storage / CDN]
● “Web consumption” [CDN -> mobile device]
![Page 9: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/9.jpg)
Lessons learnt from VP8 -> WebP
Two main use-cases for image compression:
● “Capture” [device -> storage / CDN]
● “Web consumption” [CDN -> mobile device]
“WebP”
![Page 10: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/10.jpg)
Web image format
important peculiarities
![Page 11: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/11.jpg)
Web image format important peculiarities
● incremental decoding● memory consumption● small format overhead● interleaved chunk data for early display● efficient lossy/lossless transparency● efficient lossless coding● preview● light ‘animation’ format (!= video)● efficient in software, more than hardware
![Page 12: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/12.jpg)
Web image format important peculiarities
● incremental decoding● memory consumption● small format overhead● interleaved chunk data for early display● efficient lossy/lossless transparency● efficient lossless coding● preview● light ‘animation’ format (!= video)● efficient in software, more than hardwareW
EBP v2 !!
![Page 13: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/13.jpg)
WebP v2: experimentations
Goal:
v2 = like v1 …
“Web-consumption”, not “Capture”.
![Page 14: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/14.jpg)
WebP v2: experimentations
Goal:
v2 = like v1 … … but ‘more’.
“Web-consumption”, not “Capture”.
![Page 15: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/15.jpg)
WebP v2: experimentations
Goal:
v2 = like v1 … … but ‘more’. And speed.
“Web-consumption”, not “Capture”.
![Page 16: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/16.jpg)
WebP v2: experimentations
Goal:
v2 = like v1 … … but ‘more’. And speed. And HDR.
“Web-consumption”, not “Capture”.
![Page 17: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/17.jpg)
WebP v2: how do we improve upon v1?
What can we do differently than AV1?
![Page 18: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/18.jpg)
WebP v2: how do we improve upon v1?
● floating partitioning● small-context residual coding● non-classic residuals● custom predictors● CfL● lossy/lossless alpha● more filters● more predictors● interruptibility● custom CSP transform● ANS + adaptive multi-symbol dictionaries● tiles
![Page 19: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/19.jpg)
WebP v2: how do we improve upon v1?
● floating partitioning [wip]● small-context residual coding [go]● non-classic residuals [failed so far]● custom predictors [failed so far]● CfL [go]● lossy/lossless alpha [go]● more filters [wip]● more predictors [failed so far]● interruptibility [go]● custom CSP transform [go]● ANS + adaptive SIMD multi-symbol dictionaries [go]● tiles [go]
![Page 20: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/20.jpg)
WebP v2: how do we improve upon v1?
● floating partitioning [wip]● small-context residual coding [go]● non-classic residuals [fail]● custom predictors [fail so far]● CfL [go]● lossy/lossless alpha [go]● more filters [wip]● more predictors● interruptibility [go]● custom CSP transform [go]● ANS + adaptive multi-symbol dictionaries [go]● tiles
![Page 21: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/21.jpg)
classic AV1 block partitioning
(low quality)
![Page 22: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/22.jpg)
floating block-partitioning
![Page 23: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/23.jpg)
floating block-partitioning
Parsing order = lexicographic order
X-Y sortedBuffer = 32 px-high rolling cache (max block = 32x32)Memory = O(32 * tile_width)
1 2 3 4 5
6 7
8 9
tile width
32px
![Page 24: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/24.jpg)
floating block-partitioning
Parsing order != decoding order
Strategy: try to maximize the left-sample availability
1 1 2 2 8 3 9 9 12 10
4 5 5 4
3 6 7 7 10 12 11 11
6 8 13 13 14 14 15 15 16 16
![Page 25: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/25.jpg)
1
floating block-partitioning
Parsing order != decoding order
Strategy: try to maximize the left-sample availability
2
!!
![Page 26: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/26.jpg)
1
floating block-partitioning
Parsing order != decoding order
Strategy: try to maximize the left-sample availability
(5)
2
(3)(4)
(6)
![Page 27: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/27.jpg)
1
Parsing order != decoding order
Strategy: try to maximize the left-sample availability
4
2
!! 5
3
FLUSH!!
floating block-partitioning
![Page 28: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/28.jpg)
1
Parsing order != decoding order
Strategy: try to maximize the left-sample availability
4
2
!! 5
3 (6)(7)
floating block-partitioning
![Page 29: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/29.jpg)
1
Parsing order != decoding order
Strategy: try to maximize the left-sample availability
4
2
8
5 3 6
7
floating block-partitioning
![Page 30: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/30.jpg)
Problem:
the search space is HUGE
floating block-partitioning
![Page 31: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/31.jpg)
How to do RD-Optwith this vastsearch space??
![Page 32: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/32.jpg)
Floating partitioning algo
Algo for finding a partitioning of a 32x32 section:● use variance to label 4x4 blocks with four buckets.
Variance of input 4x4 blocks:
14.0 12.5 12.0 11.8 11.3 8.1 11.1 10.1
14.6 12.0 13.3 12.6 11.9 9.9 13.3 8.7
12.2 14.6 12.6 15.0 10.3 9.2 11.5 11.2
74.7 80.8 103.0 118.5 80.1 16.6 13.2 20.5
37.4 33.4 39.2 35.6 34.6 59.8 114.7 93.4
34.5 29.9 33.1 30.2 33.4 30.0 32.4 25.2
32.1 29.9 37.1 34.5 34.7 33.7 29.9 21.7
32.9 31.5 29.6 36.1 35.9 28.7 33.3 29.4
![Page 33: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/33.jpg)
Floating partitioning algo
Algo for finding a partitioning:● use variance to label 4x4 blocks with four buckets.
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
2 2 3 3 2 0 0 0
1 1 1 1 1 2 3 2
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 0
1 1 1 1 1 1 1 1
● lay down boxes with same labels,● starting from the largest down to the smallest (finishing fill with 4x4 boxes).
![Page 34: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/34.jpg)
Floating partitioning algo
Algo for finding a partitioning:
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
2 2 3 3 2 0 0 0
1 1 1 1 1 2 3 2
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 0
1 1 1 1 1 1 1 1
![Page 35: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/35.jpg)
Floating partition algo
Algo for finding a partitioning:
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
2 2 3 3 2 0 0 0
1 1 1 1 1 2 3 2
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 0
1 1 1 1 1 1 1 1
![Page 36: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/36.jpg)
Floating partitioning algo
● Variance isn’t necessary a good metric
● too many ‘small’ blocks for filling gaps
● so many other algos to try!
![Page 37: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/37.jpg)
Floating partitioning algo
-> Still a lot of potential
trading geometry vs residuals
![Page 38: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/38.jpg)
Residual coding
![Page 39: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/39.jpg)
Residual coding
3
1 3 -1
-2 2 -1
-6 1
4
-1 -1
-1
Bounds: use Adaptive Bit to say if the residuals are bounded in X/Y. If bounded, store bounds as range.
Residual: parse as zigzag but skip anything that is outside the box:
![Page 40: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/40.jpg)
Residual coding
3
1 3 -1
-2 2 -1
-6 1
4
-1 -1
-1 -1
EOB: Adaptive Bit, but only if we have already touched both sides of the bounding box.Only 1s after When finding a 1, ABit that indicates whether all elements after are 1s.
![Page 41: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/41.jpg)
Custom CSP transform
![Page 42: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/42.jpg)
Custom CSP transformUse PCA to tight-fit the color transform matrix.
![Page 43: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/43.jpg)
Lossy-lossless alpha mix
![Page 44: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/44.jpg)
Lossy-lossless alpha mix
![Page 45: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/45.jpg)
Lossy-lossless alpha mix
![Page 46: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/46.jpg)
218 bytes.In the header.
Triangle-based preview
![Page 48: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/48.jpg)
WebP v2:
results so far
![Page 49: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/49.jpg)
WebP v2: results so far. The Good.
![Page 50: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/50.jpg)
WebP v2: results so far. The Bad.
![Page 51: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/51.jpg)
WebP v2: results so far. The Ugly.
![Page 52: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/52.jpg)
also good
![Page 53: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/53.jpg)
Syntactic decomposition
AV1
![Page 54: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/54.jpg)
Syntactic decomposition
WP2 block size coding seems more efficient!
at the detriment ofblock header
trading geometry vs residuals!
![Page 55: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/55.jpg)
Enc Speed comparison> ./examples/rd_curve kodim19.png -nomt -av1 -jpeg -webp -ssim
# Q {size (bytes), bpp, psnr (dB), SSIM*, enc-time (sec), dec-time (sec)}# | WP2 | WebP | AV1 | JPEG 0.0 5074 0.10 27.07 6.50 1.79 0.10 5028 0.10 26.49 6.44 0.04 0.00 8305 0.17 30.15 7.98 5.28 0.02 4315 0.09 22.65 5.12 0.01 0.0012.1 5776 0.12 27.50 6.69 1.86 0.10 13026 0.27 30.42 8.10 0.04 0.00 29446 0.60 35.15 11.92 12.23 0.02 11653 0.24 28.51 7.17 0.01 0.0024.3 6834 0.14 28.24 6.99 1.81 0.09 18850 0.38 31.72 9.09 0.03 0.00 47852 0.97 37.74 14.02 18.20 0.03 19015 0.39 30.71 8.55 0.01 0.0036.4 8308 0.17 29.04 7.32 1.83 0.09 24882 0.51 32.88 10.06 0.04 0.00 54919 1.12 38.48 14.61 20.71 0.03 25183 0.51 31.94 9.38 0.01 0.0048.6 11780 0.24 30.17 7.96 1.70 0.11 31518 0.64 34.04 11.04 0.04 0.00 54919 1.12 38.48 14.61 20.71 0.04 30969 0.63 32.97 10.12 0.02 0.0060.7 17264 0.35 31.79 9.04 1.79 0.11 37818 0.77 34.99 11.79 0.04 0.00 54919 1.12 38.48 14.61 20.86 0.03 36423 0.74 33.78 10.72 0.01 0.0072.9 28386 0.58 34.12 10.80 1.92 0.10 44738 0.91 35.93 12.52 0.05 0.00 54919 1.12 38.48 14.61 20.95 0.03 46192 0.94 35.07 11.67 0.02 0.0085.0 65536 1.33 39.15 14.45 2.28 0.11 73180 1.49 38.92 14.84 0.05 0.01 54919 1.12 38.48 14.61 21.22 0.03 65399 1.33 37.25 13.18 0.02 0.00
WebP3x
jpeg= ref
AV11200x
WP2120x
![Page 57: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/57.jpg)
Conclusion
Plan for 2020:
● finalize the decoding tools for experiments
● release the code base as starting point
![Page 58: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/58.jpg)
Thanks!
Questions?
![Page 59: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/59.jpg)
Extra material
![Page 60: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/60.jpg)
incremental decoding
using fiber / coroutines to pass control aroundbetween codec and network.
![Page 61: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/61.jpg)
Not yet available dataAvailable chunk
CreateLocalContext() Yield()
Bitstream
Codec::Read(data)(main context)
Codec::Decode()(local context)
Time / CPU usage
User(calling site)
WaitForNewPacket() New data chunk WaitForNewPacket()
Give execution control
SuccessfulANSDec::
ReadNextWord()
SuccessfulANSDec::
ReadNextWord()
SuccessfulANSDec::
ReadNextWord()
BlockingANSDec::
ReadNextWord()
Output buffer
return Status::Suspended;
![Page 62: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/62.jpg)
Still not thereAvailable chunk
Resume() Yield()
Bitstream
Codec::Read(data)(main context)
Codec::Decode()(local context)
New data chunk
Was blocking,now successful
ANSDec:: ReadNextWord()
BlockingANSDec::
ReadNextWord()
Discarded data
WaitForNewPacket()
return Status::Suspended;
Time / CPU usage
SuccessfulANSDec::
ReadNextWord()
User(calling site)
Output buffer
![Page 63: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/63.jpg)
Available chunk
Resume() Close()
Bitstream
Codec::Read(data)(main context)
Codec::Decode()(local context)
New data chunk
Discarded data
OnDecodedImage()
return Status::Decoded;
Time / CPU usage
User(calling site)
Output buffer
![Page 64: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/64.jpg)
Incremental decoding
Don’t assume you have the complete data for the whole frame
one must be able to quickly suspend / resume the decoding with as few work as possible
-> check points
-> coroutines in the bit-reader’s TryReadNext()
Corollary: good decoding error trapping and reporting is critical
![Page 65: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/65.jpg)
Memory consumption
Video decoding = several buffers (Ref, Alt-ref, etc.)
WebP = O(width) memory consumption
Blit to screen ASAP
animation = 1 buffer only
![Page 66: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/66.jpg)
Hardware = difficult for images
Hardware decoding is:
● per-frame oriented, non-interruptible● tricky to re-configure● non-parallelizable● unstable, sandboxed● has transfer overhead
![Page 67: Lesson learnt from WebP. What’s next?](https://reader030.vdocuments.us/reader030/viewer/2022012018/615c652619c04825032c04bf/html5/thumbnails/67.jpg)
Hardware = difficult for images
WebP experiment with Android vp8 hardware:
only 50% faster, but a lot of extra system complexity
-> Let’s target software decoding !