sr_iccp_2012.pdf
TRANSCRIPT
![Page 1: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/1.jpg)
Single Image Super-resolution
Cs129 Computational Photography
James Hays, Brown, fall 2012 Slides from Libin “Geoffrey”
Sun and James Hays
![Page 2: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/2.jpg)
Types of Super-resolution
Multi-image (sub-pixel registration)
Single-image (Hallucination)
![Page 3: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/3.jpg)
Super-resolution Goals
• 1) Produce a detailed, realistic output image.
• 2) Be faithful to the low resolution input image.
![Page 4: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/4.jpg)
Bicubic Upsampling
• 1) Produce a detailed, realistic output image.
• 2) Be faithful to the low resolution input image.
![Page 5: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/5.jpg)
Best Scene Match
• 1) Produce a detailed, realistic output image.
• 2) Be faithful to the low resolution input image.
![Page 6: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/6.jpg)
Typical Super-resolution Method
1) Build some statistical model of the visual world.
2) Coerce an upsampled image to obey those statistics.
Methods can be divided based on the statistical model – either parametric or non-parametric (data-driven).
![Page 7: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/7.jpg)
Bicubic Upsampling
![Page 8: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/8.jpg)
Fattal, SIGGRAPH 2007
![Page 9: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/9.jpg)
Example-Based Super-Resolution. Freeman, Jones, and Pasztor. 2000
![Page 10: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/10.jpg)
Example-Based Super-Resolution. Freeman, Jones, and Pasztor. 2000
Bicubic
Super-resolution
![Page 11: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/11.jpg)
Super-resolution from
Internet-scale Scene Matching
Libin (Geoffrey) Sun, James Hays
Brown University
![Page 12: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/12.jpg)
Problem Statement single image super-resolution
We want:
- more pixels
- sharp edges
- correct textures
...
85 x 128 680 x 1024 Extremely ill-posed
![Page 13: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/13.jpg)
Why is it hard?
- mathematically ill-posed
- vision-hard
sky
mountain, rocks
snow, grass
×8
![Page 14: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/14.jpg)
Previous Work
Freeman et al, 2000
Baker&Kanade, 2002
Sun et al, 2003
Yang et al, 2008
Glasner et al, 2009
Sun&Tappen, 2010
HaCohen et al, 2010
Tappen et al, 2003
Fattal, 2007
Sun et al, 2008
![Page 15: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/15.jpg)
HaCohen et al, ICCP 2010
texture database with 13 categories, 106 images - material/texture recognition is hard - requires human intervention
- edge handling - limited categories
![Page 16: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/16.jpg)
Sun & Tappen, CVPR 2010
4000 natural images, 160,000 low/high segment pairs - hard to establish 'correct' segment correspondences
Query segment Similar segments
![Page 17: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/17.jpg)
Self-similarity Based Methods [Glasner et al, 2009]
[Freedman & Fattal, 2010]
![Page 18: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/18.jpg)
Overview and Contributions
● The first to use scene matches for SR, at extremely low-res
● Scene match statistics favored over internal statistics
● Competitive results, insertion of details, texture transitions
![Page 19: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/19.jpg)
Scale of Training Data
year
Nu
mb
er o
f im
ag
es
2000 2010 2012
Freeman et al
6
Sun & Tappen
4000
HaCohen et al
106
Ours
6.3 Million
Sun et al
16 Yang et al
30
2003 2008
![Page 20: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/20.jpg)
Scene Matching: Image-level Context
![Page 21: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/21.jpg)
Scene Matching
Object recognition [Russell et al, NIPS 2007]
[Torralba et al, CVPR 2008]
Image-based rendering [Sivic et al, CVPR 2008]
Scene parsing [Liu et al, CVPR 2009]
Event prediction [Yuen & Torralba, ECCV 2010]
Image restoration/inpainting [Hays & Efros, SIGGRAPH 2007]
[Dale et al, ICCV 2009]
[Johnson et al, TVGC 2010]
Geolocation [Hays & Efros, CVPR 2008]
Image similarity [Shrivastava et al, SIGGRAPH ASIA 2011]
![Page 22: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/22.jpg)
Scene Matching
Features: Gist
color/texton histogram
sparse BoW
geometric context
6.3
million
images
Input (low-res)
Database
Example Scene Matches
![Page 23: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/23.jpg)
How Useful are the Scene Matches?
Expressiveness and Predictive Power [Zontak & Irani 2011]
Internal Database (all scales)
Input image
(ground truth)
Internal
Database (limited)
BSD
training set
Scene
Matches
External Database [Zontak & Irani 2011]
External Database [Ours]
1
2
3
4
![Page 24: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/24.jpg)
Expressiveness
High-res
patches
high-res (ground truth)
How close is the
nearest neighbor?
Database
![Page 25: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/25.jpg)
Retrieve
kNN patches
+ Estimate high-res
Predictive Error
low/high
patch pairs
low-res (observed) error in estimated HR
patch?
Database
high-res (ground truth)
![Page 26: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/26.jpg)
Segmentation: Region-level Context
![Page 27: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/27.jpg)
Segmentation: Region-level Context - 1000 textons learned per image/scene
- Color histograms
Input
image (low res)
top 5 segment matches query
segments
![Page 28: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/28.jpg)
Optimization Framework
![Page 29: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/29.jpg)
Optimization Framework
Reconstruction
term Hallucination
term
Edge
smoothness term
Image
formation model
Pixel
candidates (data-driven)
Sparse
prior (student-t)
Greedy selection of pixel candidates
[Sun & Tappen 2010]
![Page 30: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/30.jpg)
80 test images.
![Page 31: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/31.jpg)
Bicubic ×8 Ours
Sun & Tappen, CVPR 2010 Glasner et al, ICCV 2009
![Page 32: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/32.jpg)
Bicubic ×8 Ours
Glasner et al, ICCV 2009 Sun & Tappen, CVPR 2010
![Page 33: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/33.jpg)
Bicubic ×8 Ours
Glasner et al, ICCV 2009 Sun & Tappen, CVPR 2010
![Page 34: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/34.jpg)
Bicubic ×8 Ours
Glasner et al, ICCV 2009 Sun & Tappen, CVPR 2010
![Page 35: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/35.jpg)
Failure Modes: Bad Scene Match
Input image
Top Scene Matches
![Page 36: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/36.jpg)
Failures
![Page 37: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/37.jpg)
Failure Modes: Bad Texture Transfer
Input image
Top Scene Matches
![Page 38: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/38.jpg)
![Page 39: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/39.jpg)
Evaluation
Perceptual Studies, similar to [Liu et al, 2009]
- 20 test scenes - Binary comparison: 'higher quality' - 22 participants
![Page 40: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/40.jpg)
Conclusions
● The first to use scene matches for SR, at extremely low-res
● Scene match statistics favored over internal statistics
● Competitive results, insertion of details, texture transitions
![Page 41: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/41.jpg)
![Page 42: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/42.jpg)
Thank you! And
Questions?
![Page 43: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/43.jpg)
Expressiveness 5x5 vs 9x9
![Page 44: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/44.jpg)
Predictive Error 5x5 vs 9x9
![Page 45: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/45.jpg)
Predictive Uncertainty 5x5 vs 9x9
![Page 46: sr_iccp_2012.pdf](https://reader031.vdocuments.us/reader031/viewer/2022030316/577cce5b1a28ab9e788dd857/html5/thumbnails/46.jpg)
Optimization Framework [Sun 2010]