the multiscale structure of non-differentiable image...
TRANSCRIPT
![Page 1: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/1.jpg)
The Multiscale Structure of The Multiscale Structure of Non-Differentiable Image Manifolds
Michael Wakin
El t i l E i iElectrical EngineeringColorado School of Mines
Joint work with Richard Baraniuk, Hyeokho Choi, David Donoho
![Page 2: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/2.jpg)
Models for Image Structure
• Not all N-pixel images are created equal
• Models capture concise structure– few degrees of freedom– permit effective denoising, compression, registration,
detection classification segmentation estimation detection, classification, segmentation, estimation, …
![Page 3: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/3.jpg)
Geometry: Where are the Images?
concise models low-dimensional geometry
![Page 4: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/4.jpg)
Linear Subspace Models
2D F i b i ith b dli it d ie.g., 2D Fourier basis with bandlimited images
![Page 5: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/5.jpg)
Many Image Families are Highly Nonlinear
+ =
![Page 6: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/6.jpg)
Sparse Models: Unions of Subspaces
e g wavelet bases with piecewise smooth imagese.g., wavelet bases with piecewise smooth images
What more can we say about nonlinear signal families?at o e ca e say about o ea s g a a es
![Page 7: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/7.jpg)
Manifold Models• K-dimensional parameter ∈
captures degrees of freedomi i l f RNin signal f ∈ RN
f
• Signal class F = {f : ∈ } • Signal class F = {f: ∈ } forms a K-dimensional manifold– also nonparametric collections: p
faces, handwritten digits, shape spaces, etc.
• Generally nonlinear
• Surprise: Often non-differentiable
![Page 8: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/8.jpg)
Overview
• Motivating application: parameter estimation
• Non-differentiability from edge migration
• Parameter estimation (revisited)
• Non-differentiability from edge occlusion
• Manifolds in Compressive Sensing
![Page 9: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/9.jpg)
Application: Parameter Estimation
• Given an observed image I = f, g f,can we recover the underlying articulation parameters
(?,?)
articulation parameters – efficiently, and – with high precision?g p
• Given a noisy image I ≈ f, can we do the y g f,same?
• Relevant in pose estimation, image registration, computer vision, edge detection, …
![Page 10: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/10.jpg)
Newton’s Method
• Optimization problem
For a differentiable manifold project onto • For a differentiable manifold, project onto tangent planes
tangent
![Page 11: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/11.jpg)
Newton’s Method…
![Page 12: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/12.jpg)
![Page 13: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/13.jpg)
![Page 14: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/14.jpg)
![Page 15: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/15.jpg)
![Page 16: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/16.jpg)
![Page 17: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/17.jpg)
Overview
• Motivating application: parameter estimation
• Non-differentiability from edge migration
• Parameter estimation (revisited)
• Non-differentiability from edge occlusion
• Manifolds in Compressive Sensing
![Page 18: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/18.jpg)
Non-differentiability from Edge Migration
• Problem: movement of sharp edges– example: shifted disk [Donoho,Grimes]example: shifted disk [Donoho,Grimes]
• Tangents do not exist
• Visualization: Local PCA experiment
![Page 19: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/19.jpg)
![Page 20: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/20.jpg)
“Local PCA” to approximatetangent spacetangent space
![Page 21: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/21.jpg)
spanspan
![Page 22: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/22.jpg)
![Page 23: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/23.jpg)
Multiscale Tangent Structure• Family of approximate tangent planes
– T(,) scale, location on manifold
• If manifold F were differentiable:
• Does not happen when edges exist:
300 300
• Tangent spaces do not converge– twisting into new dimensionstwisting into new dimensions
• But we can study and exploit this multiscale structure l t f diff ti bl f ti~ wavelets for non-differentiable functions
![Page 24: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/24.jpg)
Shortcut to Multiscale Structurevia Regularization
• Smoothing the images smoothes the manifold
via Regularization
– more smoothing gives smoother manifold
• Example: convolution with Gaussian, width s
• Alternate family of multiscale tangent planes• Alternate family of multiscale tangent planes– tangent planes well defined, analogous to PCA
![Page 25: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/25.jpg)
Wavelet-like Characterization
• Family T(s,) like continuous wavelet transform – discretization: T(si,j) (i j)discretization: T(si,j) (i,j)
) (
log)
Fixed angle of twistb t l
scal
e between samples
space
s
• Sampling is manifold-dependent
![Page 26: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/26.jpg)
Overview
• Motivating application: parameter estimation
• Non-differentiability from edge migration
• Parameter estimation (revisited)
• Non-differentiability from edge occlusion
• Manifolds in Compressive Sensing
![Page 27: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/27.jpg)
Recall Newton’s Method…
![Page 28: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/28.jpg)
Recall Newton’s Method…
![Page 29: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/29.jpg)
Multiscale Newton Algorithm
• Construct a coarse-to-fine sequence {Fs} of manifolds that converge to F
• Take one Newton step at each scale
![Page 30: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/30.jpg)
![Page 31: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/31.jpg)
![Page 32: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/32.jpg)
![Page 33: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/33.jpg)
![Page 34: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/34.jpg)
![Page 35: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/35.jpg)
![Page 36: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/36.jpg)
![Page 37: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/37.jpg)
![Page 38: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/38.jpg)
![Page 39: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/39.jpg)
New Perspective on Multiscale Techniques
• Image Registration & • Image Registration & Coarse-to-Fine Differential Estimation– Irani/Peleg,– Belhumeur/Hager,– Keller/Averbach,Keller/Averbach,– Simoncelli– & many others…all suggested by the …all suggested by the
geometry of the manifoldmanifold
![Page 40: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/40.jpg)
Experiments: Translating Disk
![Page 41: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/41.jpg)
s = 1/2
![Page 42: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/42.jpg)
s = 1/4
![Page 43: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/43.jpg)
s = 1/16
![Page 44: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/44.jpg)
s = 1/256
![Page 45: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/45.jpg)
MSE
iterationiteration
![Page 46: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/46.jpg)
Experiments: Rotating 3-D Cube
![Page 47: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/47.jpg)
s = 1/2
![Page 48: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/48.jpg)
s = 1/4
![Page 49: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/49.jpg)
s = 1/16
![Page 50: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/50.jpg)
s = 1/256
![Page 51: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/51.jpg)
![Page 52: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/52.jpg)
Overview
• Motivating application: parameter estimation
• Non-differentiability from edge migration
• Parameter estimation (revisited)
• Non-differentiability from edge occlusion
• Manifolds in Compressive Sensing
![Page 53: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/53.jpg)
Occlusion-based Non-differentiability• Sudden appearance/disappearance of edges• Tangent spaces changing dimensiong p g g
– different “left”, “right” tangents• Occurs at every scale
pitch, roll, yaw
![Page 54: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/54.jpg)
Occlusion-based Non-differentiability
![Page 55: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/55.jpg)
Occlusion-based Non-differentiability
![Page 56: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/56.jpg)
Occlusion-based Non-differentiability
![Page 57: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/57.jpg)
Occlusion-based Non-differentiability
![Page 58: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/58.jpg)
Occlusion-based Non-differentiability
![Page 59: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/59.jpg)
Occlusion-based Non-differentiability
![Page 60: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/60.jpg)
Occlusion-based Non-differentiability
![Page 61: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/61.jpg)
Occlusion-based Non-differentiability
![Page 62: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/62.jpg)
Occlusion-based Non-differentiability
![Page 63: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/63.jpg)
Occlusion-based Non-differentiability
![Page 64: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/64.jpg)
Occlusion-based Non-differentiability
![Page 65: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/65.jpg)
Five Relevant Tangent Vectorsleft right
“head-on” view roll (1)
pitch (2)
(2)yaw (2)
• Can explicitly consider such points in parameter estimation
![Page 66: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/66.jpg)
Overview
• Motivating application: parameter estimation
• Non-differentiability from edge migration
• Parameter estimation (revisited)
• Non-differentiability from edge occlusion
• Manifolds in Compressive Sensing
![Page 67: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/67.jpg)
Compressive Sensing• Signal x is K-sparse in basis/dictionary Ψ• Collect linear measurements y = Φx
– measurement operator Φ incoherent with elements from Ψ– not adapted to signal x – random Φ will work
signal or imagemeasurements
[Candès, Romberg, Tao; Donoho]
![Page 68: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/68.jpg)
“Single Pixel” CS Camera
single photon detector
random patternon DMD array
4096 pixels4096 pixels1600 measurements
(40%)
[with R. Baraniuk + Rice CS Team]
![Page 69: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/69.jpg)
Why CS Works: Stable Embeddings
![Page 70: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/70.jpg)
One Challenge: Non-Differentiability• Many image manifolds are
non-differentiable– no embedding guarantee– difficult to navigate
• Solution: multiscale random projections
• Noiselets [Coifman et al.]
![Page 71: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/71.jpg)
Example: Ellipse Parameters
N 128 128 16384
original initial guess initial error
N = 128x128 = 16384K = 5 (major & minor axes; rotation; up & down)
M = 6 per scale (30 total): 57% success
M = 20 per scale (100 total): 99% success
![Page 72: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/72.jpg)
Multi-Signal Recovery: “Manifold Lifting”
?200 images
N = 642 = 4096N = 64 = 4096
![Page 73: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/73.jpg)
Final Reconstructionjoint reconstruction
using manifold structureimage-by-image reconstructionwithout using manifold structure
PSNR 23.8dBPSNR 15.4dB
![Page 74: The Multiscale Structure of Non-Differentiable Image Manifoldsinside.mines.edu/~mwakin/talks/cosi-mbw-web.pdf · The Multiscale Structure of Non-Differentiable Image Manifolds Michael](https://reader033.vdocuments.us/reader033/viewer/2022053006/5f098d187e708231d4275e72/html5/thumbnails/74.jpg)
Conclusions
• Image manifolds contain rich geometric structure
• Image appearance manifolds– non-differentiable, due to sharp edges– edge migration → global non-differentiability
• wavelet-like multiscale structure• accessible by regularizing each image• accessible by regularizing each image
– edge occlusion → local non-differentiability
• Can exploit multiscale structure in algorithms– proxy for standard calculus
new interpretation for image registration etc– new interpretation for image registration, etc.– applications in Compressive Sensing