image processing using partial di erential equations...
TRANSCRIPT
![Page 1: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/1.jpg)
Image Processing usingPartial Differential Equations (PDE)
Restoration, segmentation, tracking, optical flow estimation, registration
Pierre Kornprobst
NeuroMathComp project teamINRIA Sophia Antipolis - Mediterranee
Vision Student Talks [ViST]
April 2014
Kornprobst (INRIA) PDEs April 2014 1 / 48
![Page 2: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/2.jpg)
My goals today
Introduce methodology
Show links between formulations (min, ∂.∂t ,∫∫
)
Show a success story: level-sets
Kornprobst (INRIA) PDEs April 2014 2 / 48
![Page 3: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/3.jpg)
My goals today
Introduce methodology
Show links between formulations (min, ∂.∂t ,∫∫
)
Show a success story: level-sets
Kornprobst (INRIA) PDEs April 2014 2 / 48
![Page 4: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/4.jpg)
My goals today
Introduce methodology
Show links between formulations (min, ∂.∂t ,∫∫
)
Show a success story: level-sets
Kornprobst (INRIA) PDEs April 2014 2 / 48
![Page 5: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/5.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 3 / 48
![Page 6: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/6.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 3 / 48
![Page 7: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/7.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 3 / 48
![Page 8: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/8.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 3 / 48
![Page 9: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/9.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 4 / 48
![Page 10: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/10.jpg)
An image is seen as a function defined in continuous space
u : Ω ⊂ IR2 → IR
Kornprobst (INRIA) PDEs April 2014 5 / 48
![Page 11: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/11.jpg)
A PDE defines an evolution
u(t, x) /∂u
∂t= H(t, x , u,∇u,∇2u) v(x) ≡ u(∞, x)
Kornprobst (INRIA) PDEs April 2014 6 / 48
![Page 12: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/12.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 7 / 48
![Page 13: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/13.jpg)
Gaussian filtering
Let u0 an image, we define :
uσ(x) = (Gσ ∗ u0)(x) avec Gσ(x) =1
2π σ2exp
(− |x |
2
2σ2
).
σ = 0 σ = 5 σ = 11 σ = 17
Kornprobst (INRIA) PDEs April 2014 8 / 48
![Page 14: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/14.jpg)
Heat equation
A linear PDE ∂u∂t (t, x) = ∆u(t, x), t ≥ 0,
u(0, x) = u0(x).
t = 0 t = 12.5 t = 60.5 t = 93.5
A notion of scale
Kornprobst (INRIA) PDEs April 2014 9 / 48
![Page 15: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/15.jpg)
Solution of the heat equation is a convolutionu(t, x) = (G√
2 t ∗ u0)(x)
Gaussian filteringOne operation in a large neighbourhood
σ = 0 σ = 5 σ = 11 σ = 17
t = 0 t = 12.5 t = 60.5 t = 93.5
Heat equationA succession of local operations
Kornprobst (INRIA) PDEs April 2014 10 / 48
![Page 16: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/16.jpg)
Take home messages
PDE appear as a natural way to smooth images.
When it is linear, a PDE (or equivalently the convolution) do notpreserve edges.
Kornprobst (INRIA) PDEs April 2014 11 / 48
![Page 17: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/17.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 12 / 48
![Page 18: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/18.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 13 / 48
![Page 19: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/19.jpg)
Solution 1: Make convolution ”nonlinear”Bilateral filtering (Tomasi, Manduchi [1998])
Given u0, we define u by :
u(x) =1
w(x)
∫∫d(x − ξ)d(u0(x)− u0(ξ))u0(ξ)dξ with
w(x) =
∫∫d(x − ξ)d(u0(x)− u0(ξ))dξ
Kornprobst (INRIA) PDEs April 2014 14 / 48
![Page 20: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/20.jpg)
Denoising and Simplification
before after
Computer Graphics community like it!
Numerous improvements, extensions, efficient implementations and greatapplications
Kornprobst (INRIA) PDEs April 2014 15 / 48
![Page 21: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/21.jpg)
Denoising and Simplification
before after
Computer Graphics community like it!
Numerous improvements, extensions, efficient implementations and greatapplications
Kornprobst (INRIA) PDEs April 2014 15 / 48
![Page 22: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/22.jpg)
Do your own comicsWinnemoller, Olsen, Gooch [2006]
before after(video-spiderman.avi)
Kornprobst (INRIA) PDEs April 2014 16 / 48
![Page 23: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/23.jpg)
Tone management for photographic lookBae, Paris, Durand [2006]
”Clearing winter storm”, Ansel Adams
Kornprobst (INRIA) PDEs April 2014 17 / 48
![Page 24: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/24.jpg)
Tone management for photographic lookBae, Paris, Durand [2006]
”Clearing winter storm”, Ansel Adams
Kornprobst (INRIA) PDEs April 2014 17 / 48
![Page 25: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/25.jpg)
Tone management for photographic lookBae, Paris, Durand [2006]
Our classical picture
Kornprobst (INRIA) PDEs April 2014 18 / 48
![Page 26: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/26.jpg)
Tone management for photographic lookBae, Paris, Durand [2006]
model before
Kornprobst (INRIA) PDEs April 2014 19 / 48
![Page 27: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/27.jpg)
Tone management for photographic lookBae, Paris, Durand [2006]
model after
Kornprobst (INRIA) PDEs April 2014 20 / 48
![Page 28: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/28.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 21 / 48
![Page 29: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/29.jpg)
Solution 2: Modify the heat equation
Heat equation∂u
∂t= ∆u = div( ? ∇u)
Perona and Malik model [1990]
∂u
∂t= div (c(|∇u|2)
scalar∇u) with c(s) =
1/√
1 + sexp(−s)
Buades, Coll et Morel [2005]: Bilateral filter is related to Perona andMalik model
Kornprobst (INRIA) PDEs April 2014 22 / 48
![Page 30: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/30.jpg)
Solution 2: Modify the heat equation
Heat equation∂u
∂t= ∆u = div( ? ∇u)
Perona and Malik model [1990]
∂u
∂t= div (c(|∇u|2)
scalar∇u) with c(s) =
1/√
1 + sexp(−s)
Buades, Coll et Morel [2005]: Bilateral filter is related to Perona andMalik model
Kornprobst (INRIA) PDEs April 2014 22 / 48
![Page 31: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/31.jpg)
Solution 2: Modify the heat equation
Heat equation∂u
∂t= ∆u = div( ? ∇u)
Perona and Malik model [1990]
∂u
∂t= div (c(|∇u|2)
scalar∇u) with c(s) =
1/√
1 + sexp(−s)
Buades, Coll et Morel [2005]: Bilateral filter is related to Perona andMalik model
Kornprobst (INRIA) PDEs April 2014 22 / 48
![Page 32: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/32.jpg)
Diffusion acts on isophotes
Kornprobst (INRIA) PDEs April 2014 23 / 48
![Page 33: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/33.jpg)
Diffusion can interpreted w.r.t. local image structures
Nonlinear diffusion is non only a ”controlled” diffusion but it is alsorelated to a directional diffusion depending on local image structures
Most diffusion operators can be rewritten as:(. . .)
uTT +(. . .)
uNN
Kornprobst (INRIA) PDEs April 2014 24 / 48
![Page 34: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/34.jpg)
How to better take into account local image structure?
Limitation of previous models: Estimation of T ,N is local
Solution: Use structure tensor
kρ ∗ ∇uσ∇utσ = kρ ∗
(uσxx uσxyuσxy uσyy
)
Weickert [1996]
∂u
∂t= div (D(kρ ∗ ∇uσ∇ut
σ)matrix
∇u)
Kornprobst (INRIA) PDEs April 2014 25 / 48
![Page 35: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/35.jpg)
How to better take into account local image structure?
Limitation of previous models: Estimation of T ,N is local
Solution: Use structure tensor
kρ ∗ ∇uσ∇utσ = kρ ∗
(uσxx uσxyuσxy uσyy
)
Weickert [1996]
∂u
∂t= div (D(kρ ∗ ∇uσ∇ut
σ)matrix
∇u)
Kornprobst (INRIA) PDEs April 2014 25 / 48
![Page 36: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/36.jpg)
How to better take into account local image structure?
Limitation of previous models: Estimation of T ,N is local
Solution: Use structure tensor
kρ ∗ ∇uσ∇utσ = kρ ∗
(uσxx uσxyuσxy uσyy
)
Weickert [1996]
∂u
∂t= div (D(kρ ∗ ∇uσ∇ut
σ)matrix
∇u)
Kornprobst (INRIA) PDEs April 2014 25 / 48
![Page 37: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/37.jpg)
Example
Original Chaleur Perona-Malik Weickert
Kornprobst (INRIA) PDEs April 2014 26 / 48
![Page 38: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/38.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 27 / 48
![Page 39: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/39.jpg)
When a PDE is a gradient descent of an optimisationproblem (also called variational problem)
v(x) = Argminu(x)
E (u) =
∫Ω
F (x , u,∇u,∇2u)dx
Kornprobst (INRIA) PDEs April 2014 28 / 48
![Page 40: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/40.jpg)
Solution 3: Define an optimisation problem
Let us start from a model of formation of images, for example :
u0 = R u + η
where η is a white Gaussian noise and R is a linear operator.
Solve the least-square problem
infu
∫Ω|u0 − Ru|2 dx → R∗u0 − R∗Ru = 0
is an ill-posed problem
Why? Opertor R∗R is usually hard to invert (not bijective or loweingenvalues)
Kornprobst (INRIA) PDEs April 2014 29 / 48
![Page 41: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/41.jpg)
Solution 3: Define an optimisation problem
Let us start from a model of formation of images, for example :
u0 = R u + η
where η is a white Gaussian noise and R is a linear operator.
Solve the least-square problem
infu
∫Ω|u0 − Ru|2 dx → R∗u0 − R∗Ru = 0
is an ill-posed problem
Why? Opertor R∗R is usually hard to invert (not bijective or loweingenvalues)
Kornprobst (INRIA) PDEs April 2014 29 / 48
![Page 42: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/42.jpg)
Constrain possible solutions thanks to a regularityconstraint
Tikhonov et Arsenin [1977]
infu
∫Ω|u0 − Ru|2 dx + λ
∫Ω|∇u|2dx
The two main ingredients are here!
To minimise, compute the Euler-Lagrange equation, and in that case,we find again the Laplacian operator.
EL(u) = λ4u − (R∗Ru − R∗u0) = 0,∂u
∂t= EL(u).
Kornprobst (INRIA) PDEs April 2014 30 / 48
![Page 43: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/43.jpg)
Constrain possible solutions thanks to a regularityconstraint
Tikhonov et Arsenin [1977]
infu
∫Ω|u0 − Ru|2 dx + λ
∫Ω|∇u|2dx
The two main ingredients are here!
To minimise, compute the Euler-Lagrange equation, and in that case,we find again the Laplacian operator.
EL(u) = λ4u − (R∗Ru − R∗u0) = 0,∂u
∂t= EL(u).
Kornprobst (INRIA) PDEs April 2014 30 / 48
![Page 44: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/44.jpg)
Constrain possible solutions thanks to a regularityconstraint
Tikhonov et Arsenin [1977]
infu
∫Ω|u0 − Ru|2 dx + λ
∫Ω|∇u|2dx
The two main ingredients are here!
To minimise, compute the Euler-Lagrange equation, and in that case,we find again the Laplacian operator.
EL(u) = λ4u − (R∗Ru − R∗u0) = 0,∂u
∂t= EL(u).
Kornprobst (INRIA) PDEs April 2014 30 / 48
![Page 45: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/45.jpg)
So, quadratic penalty gives linear diffusion, and othewise?
Consider a general formulation :
infu
E (u) =
∫Ω
(u0 − Ru)2 dx + λ
∫Ωφ(|∇u|) dx
Diffusion operator :
div
(φ′(|∇u|)|∇u|
∇u
)=φ′(|∇u|)|∇u|
uTT + φ′′(|∇u|) uNN
Choosing φ will have consequences on the solution regularity and as aconsequence on the functional space to consider to study thevariational formulation (not commented today, but lots of interestingmaths here)
φ(s) = 2√
1 + s2 − 2 → u ∈ BV (Ω) (preserve discontinuities)
Kornprobst (INRIA) PDEs April 2014 31 / 48
![Page 46: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/46.jpg)
So, quadratic penalty gives linear diffusion, and othewise?
Consider a general formulation :
infu
E (u) =
∫Ω
(u0 − Ru)2 dx + λ
∫Ωφ(|∇u|) dx
Diffusion operator :
div
(φ′(|∇u|)|∇u|
∇u
)=φ′(|∇u|)|∇u|
uTT + φ′′(|∇u|) uNN
Choosing φ will have consequences on the solution regularity and as aconsequence on the functional space to consider to study thevariational formulation (not commented today, but lots of interestingmaths here)
φ(s) = 2√
1 + s2 − 2 → u ∈ BV (Ω) (preserve discontinuities)
Kornprobst (INRIA) PDEs April 2014 31 / 48
![Page 47: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/47.jpg)
Take home messages
PDE appear as a natural way to smooth images.
When it is linear, a PDE (or equivalently the convolution) do notpreserve edges.
Nonlinearity is needed to preserve discontinuities (seen in allformulations)
PDE may or may not derive from an optimisation problem.
Kornprobst (INRIA) PDEs April 2014 32 / 48
![Page 48: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/48.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 33 / 48
![Page 49: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/49.jpg)
A curve evolves
Lagrangian formulation∂c∂t (t, q) = v(κ, ...) N
c(0, q) = c0(q).
Example with v = κ
Kornprobst (INRIA) PDEs April 2014 34 / 48
![Page 50: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/50.jpg)
A curve evolves... but what could be the interest?
Example: Segmentation
... How to formalise the problem? By defining optimisation problems!
Kornprobst (INRIA) PDEs April 2014 35 / 48
![Page 51: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/51.jpg)
A curve evolves... but what could be the interest?
Example: Segmentation
... How to formalise the problem? By defining optimisation problems!
Kornprobst (INRIA) PDEs April 2014 35 / 48
![Page 52: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/52.jpg)
A curve evolves... but what could be the interest?
Example: Segmentation
... How to formalise the problem? By defining optimisation problems!
Kornprobst (INRIA) PDEs April 2014 35 / 48
![Page 53: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/53.jpg)
A curve evolves... but what could be the interest?
Example: Segmentation
... How to formalise the problem? By defining optimisation problems!
Kornprobst (INRIA) PDEs April 2014 35 / 48
![Page 54: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/54.jpg)
*** IF TIME *** Active contours, Kass, Witkin etal [1987]
infc
J(c) =
b∫a
∣∣ c ′(q)∣∣2dq + β
b∫a
∣∣ c ′′(q)∣∣2dq
︸ ︷︷ ︸internal energy
+ λ
b∫a
g 2( |∇I (c(q))|) dq
︸ ︷︷ ︸external energy
,
(g decreasing)
– : J(c) not intrinsic (depends on paramerisation)
– : Because of regularity constraint, topology changes are impossible(restricted to a single convex object).
– : Numerically, curve has to be initialised close to the object tosegment
Kornprobst (INRIA) PDEs April 2014 36 / 48
![Page 55: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/55.jpg)
*** IF TIME *** What about a model with curvature?
infc
J1(c) =
b∫a
∣∣c ′(q)∣∣2 dq + λ
b∫a
g 2( |∇I (c(q))| )dq
– : Still not intrinsic.
+ : No high order terms in the Euler equation.
+ : It can be shown that curvature also decreases.
Kornprobst (INRIA) PDEs April 2014 37 / 48
![Page 56: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/56.jpg)
*** IF TIME *** Idea!
Caselles, Kimmel, Kichenassamy [1995,..] :
infc
J2(c) =
b∫a
g( |∇I (c(q))| )∣∣c ′(q)
∣∣ dq.
+ : Model is intrinsic!
+ : Equivalent to infc J1, Aubert, Blanc-Ferraud [1999]
+ : Euler equation is a curve evolution :
∂c
∂t= (κg − 〈∇g ,N〉) N
Kornprobst (INRIA) PDEs April 2014 38 / 48
![Page 57: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/57.jpg)
Ok, but numerically, evolving a curve is not trivial
Distribution of points won’t stay homogeneous
Stability problems
Topology changes...
Kornprobst (INRIA) PDEs April 2014 39 / 48
![Page 58: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/58.jpg)
Idea: The Level-Sets methodDervieux, Thomasset [1980], Osher, Sethian [1988]
A curve seen as an isophote of a function
Evolve curve is equivalent to evolve function∂c∂t = v N,
c(0, q) = c0(q).=⇒
∂u∂t = v |∇u|u(0, x) = u0(x).
Kornprobst (INRIA) PDEs April 2014 40 / 48
![Page 59: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/59.jpg)
Many advantages
Fixed system of coordinates
Easy handling of topology changes
Applicable in any dimension (think about surfaces in 3D!)
Equation can be solved with suitable numerical schemes coming fromhyperbolic equations.
Kornprobst (INRIA) PDEs April 2014 41 / 48
![Page 60: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/60.jpg)
Exemple: Segmentation based on objects contoursCaselles, Kimmel, etal [1997], etc
∂u
∂t= g(|∇I |) |∇u| div
(∇u
|∇u|
)+ αg(|∇I |) |∇u|+ 〈∇g ,∇u〉
Kornprobst (INRIA) PDEs April 2014 42 / 48
![Page 61: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/61.jpg)
Exemple: Segmentation based on regionsChan, Vese [1999]
infi1,i2,c
F (i1, i2, c) = µ|c |+∫inside(c)
|u0 − i1|2dx +
∫outside(c)
|u0 − i2|2dx
Kornprobst (INRIA) PDEs April 2014 43 / 48
![Page 62: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/62.jpg)
These are just two examples...
Many other applications have been considered (e..g, textures, 3Dobjects in medical images, tracking of moving objects).
New methods to improve speed (e.g., fast marching).
Use shape priors.
Keep function as a distance function across iterations to avoidperiodic re-initialisation steps.
etc.
Kornprobst (INRIA) PDEs April 2014 44 / 48
![Page 63: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/63.jpg)
1 Can we use PDEs to do some interesting image processing?DefinitionsFrom Gaussian filtering to the heat equation
2 Three solutions to go furtherSolution 1: Make convolution ”nonlinear”Solution 2: Modify the heat equationSolution 3: Define an optimisation problem
3 A success story: Levels-sets
4 Conclusion
Kornprobst (INRIA) PDEs April 2014 45 / 48
![Page 64: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/64.jpg)
Take home messages
PDE appear as a natural way to smooth images.
When it is linear, a PDE (or equivalently the convolution) do notpreserve edges.
Nonlinearity is needed to preserve discontinuities (seen in allformulations)
PDE may or may not derive from an optimisation problem.
The notion of time evolution can be related to a notion of scale (inimage restoration) but also to different aspects like a motion (inlevel-sets)
(not shown here) Many theoretical results allow to prove if yourproblem is well defined or not.
Giving formulations in a continuous setting offers high intuitions anddiscretisation aspects only come when simulations are needed.
There has been a high activity in this area in [1990–2010] with lots ofpapers!
Kornprobst (INRIA) PDEs April 2014 46 / 48
![Page 65: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/65.jpg)
If you want to learn more
1999 2001 2002 2002, 2006, 2014
Sethian Sapiro Osher, Fedwik Aubert, Kornprobst
Kornprobst (INRIA) PDEs April 2014 47 / 48
![Page 66: Image Processing using Partial Di erential Equations …gcharpia/VisionSeminar/slides/2014-04-02... · Image Processing using Partial Di erential Equations (PDE) Restoration, segmentation,](https://reader034.vdocuments.us/reader034/viewer/2022052607/5a711fa97f8b9aac538c8ad1/html5/thumbnails/66.jpg)
Thank you!http://www-sop.inria.fr/members/Pierre.Kornprobst
Kornprobst (INRIA) PDEs April 2014 48 / 48