what is visual structure?lewicki/cp-s08/visual-structure1.pdf · is structure 2d or 3d? marr’s...
TRANSCRIPT
![Page 1: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/1.jpg)
What is visual structure?
That which allows you to identify a object?
• surface structure: shape, form, roughness, etc.
• surface properties: color, texture, material, etc.
• others?
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 1
![Page 2: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/2.jpg)
Non-locality of surface structure
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 2
![Page 3: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/3.jpg)
Non-locality of surface structure
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 3
![Page 4: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/4.jpg)
Artificial vision problems
Computation of
• features (edges, blobs, junctions, etc)
• feature classes (e.g. orientation, depth, illumination, and reflectance edges or L-,arrow-, Y-, and T- vertices)
• motion fields and optical flow
• depth maps
• lightness and color maps
• texture maps
• binocular correspondence
• image segmentation
• figure/ground organization
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 4
![Page 5: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/5.jpg)
The problem of mid-level perception
visual
Input
motion flow fields
surface color &
lightness
surface shape & structure
sensory coding
object properties &
position
visual scene structure ?
?
?
object recognition
scene segmentation
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 5
![Page 6: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/6.jpg)
Relevant structure depends on task
Could ask for each type of problem: When is it necessary to solve it?
You don’t need to solve the lightness problem if all you want to do is avoidobstacles. How much vision does the fly do?
Is structure 2D or 3D?Marr’s answer was 2.5.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 6
![Page 7: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/7.jpg)
Computing shape
Many of the probems in computational vision involve the inference of 3D form:
• shape from shading
• shape (structure) from motion
• shape from stereopsis
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 7
![Page 8: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/8.jpg)
Shape from shading
• We easily recognize theseas the same shape
• But: retinal images varygreatly
• Shadows and brightnessgradients provide mostshape information
• How can we recoverstructure from intensitypatterns?
• Harder problem: How dowe determine similar 3Dstructures?
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 8
![Page 9: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/9.jpg)
Luminance, irradiance, and reflectance
BRDF is ratio of radiance (reflectedlight) to irradiance (illuminant light).Using φ = (θ, ϕ)
R(φi, φe) =δL(φe)δE(φi)
In general, the luminance image is theproduct of the illumance and thereflectance
L(x, y, φe) = E(x, y, φi)R(x, y, φi, φe)
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 9
![Page 10: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/10.jpg)
Non-point light sources
How to calculate radiance in thepresence of extended light sources?
Intergrate product of source radianceand BRDF over all indicent directions:
L(φe) =∫
Ω
E(φi)R(φi, φe) cos θidφi
where dφi = sin θidθidϕi.
• cos θi accounts for forshortening ofsurface as seen from directionφi = (θi, ϕi)
• Some surfaces are complex, i.e.radiances changes with viewpoint(e.g. brushed metal).
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 10
![Page 11: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/11.jpg)
Idealized surfaces
• Lambertian: R(φi, φe) is constant (matte surfaces).
L =ρ(λ)πE0 cos θi
ρ(λ) is the albedo and describes theratio of irradiance caused byillumination and total radiancescattered back. For colored objects,this depends on wavelength λ.
• Specular: light is reflected like a mirror.
Specular direction δθ has finite width due to
– variable scatter from surface variations– non-point light sources
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 11
![Page 12: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/12.jpg)
Surface orientation
The surface normal n is the cross product of any two distinct tangents.Take rx = (1, 0, p)T and ry = (0, 1, q)T , then (if n points toward viewer)
n = rx × ry = (−p,−q, 1)T
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 12
![Page 13: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/13.jpg)
Surface normals
The unit surface normal is
n =n
|n|=
(−p,−q, 1)T√1 + p2 + q2
Now we can calculate angle θe between viewer and surface normal. Assumingviewer is at (0, 0, 1)T , then taking dot product yields
cos θe =1√
1 + p2 + q2
This relates surface orientation to the image gradient.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 13
![Page 14: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/14.jpg)
Reflectance maps
We still need the light source. Assume (−ps,−qs, 1)T points to the light source. Ifwe consider a Lambertian surface
L =1πE cos θi θi > 0
where θi is the angle between n and source, then we can compute a reflectancemap for a known light source. Taking the dot product
R(p, q) = cos θi =1 + psp+ qsq√
1 + p2 + q2√
1 + p2s + q2
s
Different assumptions (e.g. specular instead of Lambertian) require differentformulas.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 14
![Page 15: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/15.jpg)
An example reflectance map
Makes explicit the relation between surface orientation and brightness.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 15
![Page 16: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/16.jpg)
Applying the reflectance map to a surface
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 16
![Page 17: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/17.jpg)
Shaded reflectance map using lighting directions
View is looking down from above.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 17
![Page 18: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/18.jpg)
That’s graphics, but what about vision?
We go backwards. But there are many problems:
• How do we determine the lighting direction?
• What about multiple light sources, or ambient lighting?
• How do we separate shading from paint?
• How do we resolve ambiguity?
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 18
![Page 19: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/19.jpg)
A simple shading pattern has many possible interpretations
Freeman: A Bayesian approach to the generic viewpoint assumption.
• Mathematically, many shapes and lighting directions could explain the pattern• Perceptually, there only two.• How does the visual system know which shape is correct?• Shapes 2 - 4 require accidental alignment with the light source.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 19
![Page 20: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/20.jpg)
Generic viewpoint assumption
Interpreting this as cube requiresassuming an accidental view.
The generic viewpoint assumptionassumes a scene is not viewed from aspecial position.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 20
![Page 21: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/21.jpg)
Change in shading with respect to image
• shape 3: only a small range of directions are consistent with image• shape 5: much larger range of directions give same image• Equally likely light directions ⇒ shape 5 more probable than shape 3
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 21
![Page 22: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/22.jpg)
Freeman: Exploiting the Generic Viewpoint Assumption
Main points:
• quantify generic view probabilities in a general case
• generic variable can be general, e.g. object orientation or lighting position
• generic view assumption can strongly influence scene interpretations
Key is to quantify how visual data would change if the data were to change:large changes correspond to improbable scenes
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 22
![Page 23: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/23.jpg)
Scene probability equation
• Fidelity: how well rendering model can describe data
• Prior probability: how parameters fit prior expectations
• Genericity: favors interpretations for which the image is stablew.r.t changes in generic variables
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 23
![Page 24: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/24.jpg)
Different reflectances and shapes produce same image
• Small Rotations can distinguish between possibilities.• If all viewpoints, shapes, and reflectances are equally likely⇒ (b-c) more likely than (e-g)
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 24
![Page 25: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/25.jpg)
Roles reversed: Lambertian surface changes more than shiny
If all object orientations are equally likely, thenprobability of object ∝ range of angles that yield “similar” images
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 25
![Page 26: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/26.jpg)
The scene probability equation
The image y is modeled by the “rendering function” plus noise:
y = f(β, x) + n
• β: vector of scene parameters we want to estimate,e.g. the object shape and reflectance function• x: vector of generic variables we don’t want to estimate,
e.g. viewpoint, object orientation, lighting position• n ∼ Gaussian ⇒ image likelihood is Gaussian
P (y|β, x) ∼ N (y − f(β, x), σ)
Goal is to derive P (β|y), i.e. the probability of the scene parameters given theimage, independent of the generic variables.
Question:
Why do we want P (β|y) and not P (β|x, y) or P (β, x|y)?
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 26
![Page 27: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/27.jpg)
Deriving P (β|y)
Use Bayes rule:
P (β, x|y) =P (y|β, x)P (β)P (x)
P (y)
This gives joint probability of β and x given y.
P (β|y) is obtained by marginalization
P (a) =∫P (a, b)da
In the case of the image model
P (β|y) =P (β)P (y)
∫P (y|β, x)P (x)dx
How do we evaluate the integral?
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 27
![Page 28: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/28.jpg)
Approximating the integral∫P (y|β, x)P (x)dx
The image likelihood, P (y|β, x), has the form
P (y|β, x) =1Zσ
exp[−|y − f(β, x)|2
2σ2
]
Cannot solve analytically because of the form of f , so approximate:
• Taylor expand around peak of P (y|β, x),i.e. around value of x0 that minimizes |y − f(β, x)|2.• This is called Laplace’s approximation• It approximates the posterior volume with a Gaussian.
The posterior probability is then given by
P (β|y) ∝ exp[−|y − f(β, x0)|2
2σ2
]P (β)P (x0)
1√det(A)
∝ “fidelity” prior “genericity”
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 28
![Page 29: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/29.jpg)
Specifying the model
• Still need to specify f(β, x)
• Use simple models based on rendering equations
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 29
![Page 30: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/30.jpg)
Parameterization of the reflectance function
• surface roughness governs width of specular highlight• specularity determins ratio of diffuse and specular reflections
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 30
![Page 31: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/31.jpg)
Relative probabilities of surface properties
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 31
![Page 32: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/32.jpg)
Generic light direction
Linear shading (Pentland, 1990): assume that image intensities are approximatelyproportional to surface slopes p and q
I = k1p+ k2q
Related facts:
• direction of light is tan−1(k1, k2)• product of lighting strength and surface reflectance is
√k2
1 + k22
• surface slopes scale inversely with√k2
1 + k22
Any assumed lighting direction and strength can explain an image by assuming adifferent shape.
Which shape and lighting parameters are best?
• Don’t want to assume aribitrarily that some shapes are more likely.→ Derive scene probability equation for linear shading model.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 32
![Page 33: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/33.jpg)
Relative probabilities of the original bump image
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 33
![Page 34: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/34.jpg)
Proability of shape vs lighting direction
• All shapes are equally probable apriori
• Shapes reconstructed from correctlighting direction are more probablethan other directions
• can also infer correct vertical scalingand light strength
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 34
![Page 35: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/35.jpg)
The effect of the genericity term
• image with 7dB Gaussiannoise added (a)• image has two
explanations: lightingfrom left (b) and lightingfrom top (c)• side lighting better
explains image, i.e. hashigher fidelity (f)• but light source must be
very percisely positioned,so genericity of b ismuch lower than c (f)• final probability favors c
even though b is a betterfit to image data
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 35
![Page 36: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/36.jpg)
Performance of Shape from Shading Algorithms
![Page 37: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/37.jpg)
– Summary –
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 37
![Page 38: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/38.jpg)
Lambertian reflectance model
• brightness proportional to energy of incident light
• amount of light energy on surface ∝ surface area as seen from light sourceposition (foreshortening)
• Brightness of Lambertian surface is product of strengh of light source, A, thealbedo of surface, ρ, and foreshortened area, cos θi
IL ≡ R = Aρ cos θi
where R is reflectance map, θi is angle between surface normal ~N and sourcedirection ~S. For unit normals, we can write
IL = Aρ ~N · ~S
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 38
![Page 39: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/39.jpg)
Specular reflectance model
• Specularity only occurs when incident angle of light source equals reflected angle
• simplest model for the brightness of the specular reflection is
IS = Bδ(θs − 2θr)
where B is the strength of the specular component, θs is the angle between thelight source and the viewing direction, and θr is the angle between the surfacenormal and the viewing direction.
• this assumes a point reflection, which is not realistic. There are many ways tomake this more realistic, but a simple one is
IS = K exp−( αm)2
where K is a constant, α is the angle between the surface normal ~N and thebisector ~H of the viewing direction and source direction.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 39
![Page 40: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/40.jpg)
Non-ideal surfaces
• Most surfaces are not idea.
• can be approximated by hybrid surfaces:
I = (1− ω)IL + ωIs
• many more complex models exist
• Lambertian model is usually a poor approximation due to surfaces roughness
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 40
![Page 41: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/41.jpg)
Inferring surface orientation
• p and q are the surface gradientsalong the x and y direction.
• general problem is to infer p and q(or equivalently the surface normal~N = (−p,−q, 1)T ) from the observedimage intensities
• some algorithms assume knowledgeof lighting sources and viewingangles, more sophisticated algorithmsestimate these from the image
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 41
![Page 42: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/42.jpg)
Minimization approaches
• describe constraints that are consistent with reflection equations and priorknowledge
• find solution (i.e. depth and surface orientation) over whole image thatminimizes constraint equation
brightness constraint: minimize total brightness error of reconstructed image∫ ∫(I −R)2 dx dy
Intensity gradient constraint: minize error between reconstructed intensitygradient and image intensity gradient in both x and y directions∫ ∫
(Rx − Ix)2 + (Ry − Iy)2 dx dy
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 42
![Page 43: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/43.jpg)
Other constraints
Solution of brightness constraint is ill-posed, so assume smooth surface
smoothness constraint: bias toward smooth surfaces to obtain unique solution∫ ∫(p2x + p2
y + q2x + q2
y) dx dy
Integrability constraint: ensure that the surface height Z(x, y) obtained byintegrating over surface slopes is independent of the path of integration, i.e.Zx,y = Zy,x. This can be written∫ ∫
((Zx − p)2 + (Zy − q)2 dx dy
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 43
![Page 44: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/44.jpg)
Depth maps of synthetic images
Synthetic images generated using 2 different light sources:
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 44
![Page 45: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/45.jpg)
Example minimization algorithm: Zheng and Chellappa
Miminize the constraint:∫ ∫(I −R)2 + (Rx − Ix)2 + (Ry − Iy)2 + µ((Zx − p)2 + (Zy − q)2) dx dy
• this combines brightness, intensity gradient, and integrability constraints
• minimized iteratively by optimizing depth and gradients
• use hierarchical (pyramid) structure to speed up computation
• µ = 1
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 45
![Page 46: What is visual structure?lewicki/cp-s08/visual-structure1.pdf · Is structure 2D or 3D? Marr’s answer was 2.5. ... presence of extended light sources? ... w.r.t changes in generic](https://reader031.vdocuments.us/reader031/viewer/2022030506/5ab4ff157f8b9a156d8c6224/html5/thumbnails/46.jpg)
Results of minimization algorithm
(a,b) results with light source 1. (c,d) light source 2.
CP08: Visual Structure 1 / Michael S. Lewicki, CMU áá á
á
? 46