Download - Image optimization q_auto - f_auto
![Page 1: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/1.jpg)
q_auto,f_auto A peek under the hood
IMAGE COMPRESSION: GRACEFUL DEGRADATIONFOR BETTER PERFORMANCE
Jon SneyersOctober 2016
![Page 2: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/2.jpg)
AUTO EVERYTHING: G_AUTO,W_AUTO,DPR_AUTO,Q_AUTO,F_AUTO
DELIVER THE RIGHT IMAGE TO EVERYONE▸ Different crops depending on context (g_auto,w_auto)
Responsive design
Android tablet185K WebP1500x844 pixels
Firefox desktop381K JPEG
2000x1125 pixels
iPhone54K JPEG
500x500 pixels
![Page 3: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/3.jpg)
AUTO EVERYTHING: G_AUTO,W_AUTO,DPR_AUTO,Q_AUTO,F_AUTO
DELIVER THE RIGHT IMAGE TO EVERYONE
Big Android phoneHigh bandwidth
144K WebP1000x1000 pixels
Windows phone142K JPEG XR
1000x1000 pixels
iPhone54K JPEG
500x500 pixels
Retina iPhone201K JPEG
1000x1000 pixels
Small Android phoneLow bandwidth
43K WebP500x500 pixels
Topic of this webinar
▸ Same image, different encoding depending on image and context (q_auto,f_auto,dpr_auto)
![Page 4: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/4.jpg)
Q_AUTO,F_AUTO - INTRO
FIXED QUALITY, FIXED FORMAT
▸ Common practice: fixed format, fixed quality, fixed encode settings
▸ E.g. WordPress default: re-encode as JPEG quality 82
▸ E.g. Facebook, Twitter, Instagram: re-encode as low-quality JPEG
▸ This is suboptimal for three reasons:
▸ Optimal quality depends on the image: some images look fine at JPEG quality 50, others have visible compression artifacts at quality 85
▸ Optimal quality/size trade-off depends on context(viewing device, network conditions, …)
▸ Best format depends on 1) image, and 2) receiver decode capabilities
![Page 5: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/5.jpg)
Q_AUTO,F_AUTO - INTRO
INTRODUCING Q_AUTO,F_AUTO
▸ Automatic quality, automatic image format
▸ q_auto:low, q_auto:eco, q_auto:good, q_auto:best
▸ Deliver the optimal file to every browser/device
▸ Most suitable format is picked among all formats the receiver accepts (Accepts header), can depend on the image
▸ Suitable quality settings are picked automatically
▸ Depending on the image content
▸ Save-Data client hint lowers the target quality(from q_auto:good to q_auto:eco)
![Page 6: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/6.jpg)
Q_AUTO,F_AUTO - TARGET FORMATS
TARGET IMAGE FORMATS
▸ JPEG
▸ All browsers, only opaque images (no alpha channel supported)
▸ PNG
▸ All browsers, not very useful for opaque photographs
▸ WebP
▸ Chrome/Opera/Android only, useful for all kinds of images
▸ WDP (JPEG XR)
▸ IE/Edge only, mostly useful for photographs
![Page 7: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/7.jpg)
Q_AUTO,F_AUTO - TARGET FORMATS
LOSSY IMAGE COMPRESSION BASICS
▸ Luma (Y or L) and Chroma (U,V or Cb,Cr or Co,Cg or a,b)
▸ Chroma subsampling 4:4:4 vs 4:2:0
![Page 8: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/8.jpg)
Q_AUTO,F_AUTO - TARGET FORMATS
LOSSY IMAGE COMPRESSION BASICS
▸ Discrete Cosine Transform (DCT)
LOW FREQUENCY
NOT TOO LO
SSY
HIGH FREQUENCY
VERY LOSSY
![Page 9: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/9.jpg)
Q_AUTO,F_AUTO - TARGET FORMATS
JPEG ENCODE PARAMETERS TO DECIDE
▸ Quality (amount of DCT quantization)
▸ Separate quality setting for luma and chroma
▸ Chroma subsampling: 4:4:4 vs 4:2:0
▸ or “in between” (4:4:4 with lower quality for chroma than luma)
▸ MozJPEG vs libjpeg-turbo
▸ Progressive (smaller, previews) vs non-progressive (faster to decode)
▸ or “in between” (custom progressive scan script)
![Page 10: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/10.jpg)
Q_AUTO,F_AUTO - TARGET FORMATS
LOSSY PNG? SURE!
▸ PNG is a lossless format, but it can be used in a lossy way
▸ Two options:
▸ Quantize to PNG8
▸ Maximum 256 RGBA colors
▸ Can use fewer than 256 colors for better compression
▸ Apply a lossy PNG24/32 transformation (diagonal blur)
![Page 11: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/11.jpg)
Q_AUTO,F_AUTO - TARGET FORMATS
WEBP: ACTUALLY TWO IMAGE FORMATS
▸ Lossy WebP (VP8 intra-frame)
▸ Always 4:2:0 chroma subsampling (can be problematic)
▸ Usually better than JPEG but certainly not always
▸ Supports alpha
▸ Lossless WebP
▸ Mostly similar to PNG but better compression
▸ Decisions: lossy or lossless? which quality? or fall back to JPEG or PNG?
![Page 12: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/12.jpg)
Q_AUTO - ALGORITHM
THE Q_AUTO ALGORITHM: OVERVIEW
▸ Heuristics that examine the original pixels
▸ Trial encodes
▸ Perceptual metric and heuristics to score trial encodes
▸ Heuristics to select the best format and to decide the actual encode settings based on the above information
▸ Important constraint: it has to be reasonably fast if we want to deploy it at “Cloudinary scale”. Latency and cpu cost has to be reasonable.
![Page 13: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/13.jpg)
Uncompressed Image(no alpha channel)
NON-PHOTOGRAPHIC?
Photographic Non-photographic(text, line art, screenshot, …)
CHROMA SUBSAMPLING?TWO TRIAL JPEG ENCODINGS
(4:4:4 AND 4:2:0)
Y-DSSIM SCORE
Target quality(low, eco, good,best)
TRIAL LOSSLESS PNG/WEBP
LIBJPEG 4:2:0 ENCODING
PNG ENCODING (PNGQUANT)
LOSSLESS WEBP
LOSSY WEBP
ANALYSIS ADJUSTING QUALITY
ENCODING
LIBJPEG 4:4:4 ENCODING
MOZJPEG 4:2:0 ENCODING
Too large
compared to
trial JPEG
very bad score:encode another trial JPEGat higher quality
Few colors
![Page 14: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/14.jpg)
Uncompressed Imagewith alpha channel
NON-PHOTOGRAPHIC?
Non-photographic(text, line art, screenshot, …)
CHROMA SUBSAMPLING?TWO TRIAL JPEG ENCODINGS
(4:4:4 AND 4:2:0)
Y-DSSIM SCORE
TRIAL LOSSLESS WEBP
LIBJPEG 4:2:0 ENCODING
PNG ENCODING (PNGQUANT)
LOSSLESS WEBP
LOSSY WEBP
ANALYSIS ADJUSTING QUALITY
ENCODING
LIBJPEG 4:4:4 ENCODING
MOZJPEG 4:2:0 ENCODING
Photographic
Too large
compared to
trial JPEG
Possible, but not usefulif WebP can be used
JPEG does not support alpha
Target quality(low, eco, good,best)
Uncompressed Imagewith alpha channel
very bad score:encode another trial JPEGat higher quality
![Page 15: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/15.jpg)
Uncompressed Imagewith alpha channel
LIBJPEG 4:2:0 ENCODING
PNG ENCODING (PNGQUANT)
LOSSLESS WEBP
LOSSY WEBP
ANALYSIS ADJUSTING QUALITY
ENCODING
LIBJPEG 4:4:4 ENCODING
MOZJPEG 4:2:0 ENCODING
Target quality(low, eco, good,best)
![Page 16: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/16.jpg)
Q_AUTO - ALGORITHM
PHOTOGRAPHIC OR NON-PHOTOGRAPHIC?
Original: 634K PNG(lossless)
71K JPEG(lossy)
204K PNG(lossy)👍 👎
![Page 17: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/17.jpg)
Q_AUTO - ALGORITHM
PHOTOGRAPHIC OR NON-PHOTOGRAPHIC?
Original: 221K PNG(lossless)
97K JPEG(lossy)
50K PNG(lossy) 👍👎
![Page 18: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/18.jpg)
Q_AUTO - ALGORITHM
CHROMA SUBSAMPLING?
▸ Compute a map of “chroma-related compression artifacts”
▸ A lot of artifacts? Use 4:4:4 with high chroma quality
▸ Don’t use lossy WebP in this case
▸ Some artifacts? Use 4:4:4 with lower chroma quality
▸ (Almost) no artifacts? Use 4:2:0NON-PHOTOGRAPHIC?
CHROMA SUBSAMPLING?
Y-DSSIM SCORE
TRIAL LOSSLESS PNG/WEBP
LIBJPEG PNGLOSSLESS WEBP
LOSSYWEBPMOZJPEG
![Page 19: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/19.jpg)
Q_AUTO - ALGORITHM
CHROMA SUBSAMPLING?
Original: 505K PNG(lossless)
65K JPEG(q_75, 4:4:4)
57K JPEG(q_75, 4:2:0)👍👎
![Page 20: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/20.jpg)
Q_AUTO - ALGORITHM
CHROMA SUBSAMPLING?
Original image(lossless)
4:4:4 JPEG(lossy, q_100 and q_75)
4:2:0 JPEG(lossy, q_100 and q_75)😱
![Page 21: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/21.jpg)
Q_AUTO - ALGORITHM
PERCEPTUAL METRIC: Y-DSSIM
▸ Compute multi-scale structural similarity metric (SSIM) between uncompressed and trial JPEG
▸ Adjust quality accordingly (repeat if needed)
▸ Only look at luma (Y)
▸ No need to look at chroma, chroma subsampling test already covers that
NON-PHOTOGRAPHIC?
CHROMA SUBSAMPLING?
Y-DSSIM SCORE
TRIAL LOSSLESS PNG/WEBP
LIBJPEG PNGLOSSLESS WEBP
LOSSYWEBPMOZJPEG
![Page 22: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/22.jpg)
Q_AUTO - JPEG ENCODING
ACTUAL JPEG ENCODE
▸ Use either MozJPEG or libjpeg-turbo (depending on quality parameters)
▸ Almost always use 5-scan “semi-progressive” scan script
▸ Faster encode/decode than default (~10 scans) progressive, still most of the benefits
▸ Only use non-progressive for
▸ very small images(no advantage)
▸ very large images (receiver might not have the memory)
NON-PHOTOGRAPHIC?
CHROMA SUBSAMPLING?
Y-DSSIM SCORE
TRIAL LOSSLESS PNG/WEBP
LIBJPEG PNGLOSSLESS WEBP
LOSSYWEBPMOZJPEG
![Page 23: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/23.jpg)
Q_AUTO - JPEG ENCODING
JPEG - DECODE TIME
▸ JPEG decode speed:
▸ Non-progressive: ~215 MP/s
▸ Default progressive: ~111 MP/s
▸ Semi-progressive: ~184 MP/s
▸ Total decode time has a (minor) effect on time-to-render, in particular if the image is cached locally
▸ If transfer is the bottleneck, (semi-)progressive has the advantage of progressive previews and smaller total size
![Page 24: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/24.jpg)
Q_AUTO - PNG ENCODING
ACTUAL PNG ENCODE
▸ Use pngquant to quantize colors & improve compression
▸ Use PNG8 if the error is not too bad
▸ The number of colors is picked to reach a specific MSE
▸ Otherwise use lossy PNG24/32 (rarely needed)
▸ Quantize not the pixel values, but the filtered pixel values w.r.t. the ‘average’ predictor ((left+up)/2)
▸ Slight diagonal blur,no quantization/dithering artifacts
▸ Then use optipng to optimize further
NON-PHOTOGRAPHIC?
CHROMA SUBSAMPLING?
Y-DSSIM SCORE
TRIAL LOSSLESS PNG/WEBP
LIBJPEG PNGLOSSLESS WEBP
LOSSYWEBPMOZJPEG
![Page 25: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/25.jpg)
Q_AUTO - PNG ENCODING
COLOR QUANTIZATIONOriginal PNG24, 573KB 256 colors, 203KB 100 colors, 148KB
30 colors, 98KB 5 colors, 39KB 3 colors, 24KB
![Page 26: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/26.jpg)
Q_AUTO - PNG ENCODING
SOMETIMES PNG8 IS TOO LOSSY
▸ Original: 88,358 bytes
▸ PNG8: 21,739 bytes
▸ Too lossy: banding / visible dither
▸ Lossy PNG32: 57,552 bytes
▸ 53,101 bytes at q_auto:eco
▸ 64,771 bytes at q_auto:best
![Page 27: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/27.jpg)
Q_AUTO - PNG ENCODING
SOMETIMES PNG8 IS TOO LOSSY▸ Original: 88,358 bytes
▸ PNG8: 21,739 bytes
▸ Lossy PNG32: 53,101 bytes
![Page 28: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/28.jpg)
EVALUATIONQ_AUTO
![Page 29: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/29.jpg)
Q_AUTO - EVALUATION
EVALUATION: A DELICATE TRADE-OFF
▸ File size (bandwidth/storage cost)
▸ Perceptual quality compared to original
▸ Encode time (cpu cost, latency of new images)
▸ End-user experience:
▸ time-to-render (includes latency, download, decode, progressive rendering)
▸ perceptual quality with no access to original (e.g. blur vs blockiness, etc.)
![Page 30: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/30.jpg)
Q_AUTO - EVALUATION
VISUAL INSPECTION
![Page 31: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/31.jpg)
Q_AUTO - EVALUATION
VISUAL INSPECTION
![Page 32: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/32.jpg)
Q_AUTO - EVALUATION
VISUAL INSPECTION
![Page 33: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/33.jpg)
Q_AUTO - EVALUATION
VISUAL INSPECTION
![Page 34: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/34.jpg)
Q_AUTO - EVALUATION
VISUAL INSPECTION
![Page 35: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/35.jpg)
Q_AUTO - EVALUATION
VISUAL INSPECTION
![Page 36: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/36.jpg)
Q_AUTO - EVALUATION
PERCEPTUAL METRICS
▸ Commonly used simple metrics are not really good enough (e.g. RGB-PSNR)
▸ Perceptually uniform color space: RGB < YCbCr < HSL < L*a*b*
▸ Kornel Lesiński’s DSSIM: L*a*b* multi-scale SSIM
▸ Our own metric: a variant of L*a*b* multi-scale SSIM with some improvements
![Page 37: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/37.jpg)
Q_AUTO - EVALUATION - PHOTOGRAPHIC IMAGES
CLOUDINARY_DSSIM (MOST ACCURATE)
Y-DSSIM (USED IN Q_AUTO)
KORNEL’S DSSIM
ARTIFACT DANGER ZONE
![Page 38: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/38.jpg)
Q_AUTO - EVALUATION - NON-PHOTOGRAPHIC IMAGES
![Page 39: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/39.jpg)
Q_AUTO,F_AUTO - CONCLUSION
CONCLUSION
▸ Using q_auto,f_auto ensures that every image gets encoded at the right quality, in the best available format
▸ No visible compression artifacts
▸ No wasted bandwidth
▸ Less bandwidth needed on average (though not for every individual image: the q_auto quality might be higher than the default quality)
![Page 40: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/40.jpg)
UPCOMING CLOUDINARY WEBINARS
Responsive Images: Past, Present & Future Wednesday Nov 16th , 11am PT
Image Management: Buy vs. Build, Build.comWednesday Dec 7th , 11am PT
Don’t fall behind. Check upcoming webinars: http://info.cloudinary.com/webinars.html
![Page 41: Image optimization q_auto - f_auto](https://reader031.vdocuments.us/reader031/viewer/2022021420/5886cfbe1a28abcc7d8b8145/html5/thumbnails/41.jpg)
QUESTIONS?q_auto,f_auto
Uncompressed Image
NON-PHOTOGRAPHIC?
Photographic Non-photographic
CHROMA SUBSAMPLING?TWO TRIAL JPEG ENCODINGS
(4:4:4 AND 4:2:0)
Y-DSSIM SCORE
Target quality
TRIAL LOSSLESS PNG/WEBP
LIBJPEG 4:2:0
PNG ENCODING (PNGQUANT)
LOSSLESS WEBP
LOSSY WEBP
LIBJPEG 4:4:4
MOZJPEG 4:2:0