![Page 1: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/1.jpg)
Learning to separate shading from paint
Marshall F. Tappen1
William T. Freeman1
Edward H. Adelson1,2
1MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)
2MIT Dept. of Brain and Cognitive SciencesMarshall
![Page 2: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/2.jpg)
Forming an Image
Surface
Illuminate the surface to get:
Shading ImageThe “shading image” is the interaction of the shapeof the surface and the illumination
![Page 3: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/3.jpg)
Painting the Surface
Scene ImageWe can also include a reflectance pattern or a “paint”image. Now shading and reflectance effects combine to create the observed image.
![Page 4: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/4.jpg)
ProblemHow can we access shape or reflectance information from the observed image?For example:
image estimate of shape
![Page 5: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/5.jpg)
Goal: decompose the image into shading and reflectance components.
=
Image Shading Image Reflectance Image
• These types of images are known as intrinsic images (Barrow andTenenbaum).
• Note: while the images multiply, we work in a gamma-corrected domain and assume the images add.
![Page 6: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/6.jpg)
Why you might want to compute these intrinsic images
• Ability to reason about shading and reflectance independently is necessary for most image understanding tasks.– Material recognition– Image segmentation
• Want to understand how humans might do the task.• An engineering application: for image editing, want
access and modify the intrinsic images separately• Intrinsic images are a convenient representation.
– More informative than just the image– Less complex than fully reconstructing the scene
![Page 7: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/7.jpg)
Treat the separation as a labeling problem
• We want to identify what parts of the image were caused by shape changes and what parts were caused by paint changes.
• But how represent that? Can’t label pixels of the image as “shading” or “paint”.
• Solution: we’ll label gradients in the image as being caused by shading or paint.
• Assume that image gradients have only one cause.
![Page 8: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/8.jpg)
Recovering Intrinsic Images• Classify each x and y image derivative as being
caused by either shading or a reflectance change• Recover the intrinsic images by finding the least-
squares reconstruction from each set of labeled derivatives. (Fast Matlab code for that available from Yair Weiss’s web page.)
Original x derivative image Classify each derivative(White is reflectance)
![Page 9: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/9.jpg)
Classic algorithm: Retinex
• Assume world is made up of Mondrian reflectance patterns and smooth illumination
• Can classify derivatives by the magnitude of the derivative
![Page 10: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/10.jpg)
Outline of our algorithm (and the rest of the talk)
• Gather local evidence for shading or reflectance– Color (chromaticity changes)– Form (local image patterns)
• Integrate the local evidence across space.– Assume a probabilistic model and use belief
propagation.• Show results on example images
![Page 11: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/11.jpg)
Probabilistic graphical model
UnknownDerivative Labels(hidden random variables that we want to estimate)
![Page 12: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/12.jpg)
Probabilistic graphical model• Local evidence
Local Color Evidence
Some statistical relationship that we’ll specify
Derivative Labels
![Page 13: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/13.jpg)
Probabilistic graphical model• Local evidence
Derivative Labels
Local Color EvidenceLocal Form Evidence
![Page 14: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/14.jpg)
Probabilistic graphical modelPropagate the local evidence in Markov Random Field. This strategy can be used to solve other low-level vision problems.
Local Evidence
Hidden state to be estimated
Influence of Neighbor
![Page 15: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/15.jpg)
Local Color Evidence
For a Lambertian surface, and simple illumination conditions, shading only affects the intensity of the color of a surface
Notice that the chromaticity of each face is the same
Any change in chromaticity must be a reflectance change
![Page 16: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/16.jpg)
Classifying Color ChangesIntensity ChangesChromaticity Changes
Angle between the two vectors, θ, is greater than 0
Angle between two vectors, θ,equals 0
Red
Green
Blue
Red
GreenBlue
θ
![Page 17: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/17.jpg)
1. Normalize the two color vectors c1and c2
2. If (c1 c2) > T• Derivative is a reflectance change• Otherwise, label derivative as shading
Color Classification Algorithm
c1 c2
![Page 18: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/18.jpg)
Result using only color information
![Page 19: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/19.jpg)
Results Using Only Color
Input ReflectanceShading• Some changes are ambiguous• Intensity changes could be caused by shading or
reflectance– So we label it as “ambiguous”– Need more information
![Page 20: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/20.jpg)
Utilizing local intensity patterns
• The painted eye and the ripples of the fabric have very different appearances
• Can learn classifiers which take advantage of these differences
![Page 21: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/21.jpg)
Shading/paint training set
Examples from Reflectance Change Training Set
Examples from Shading Training Set
![Page 22: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/22.jpg)
From Weak to Strong Classifiers: Boosting
• Individually these weak classifiers aren’t very good.• Can be combined into a single strong classifier.• Call the classification from a weak classifier hi(x).• Each hi(x) votes for the classification of x (-1 or 1).• Those votes are weighted and combined to produce a
final classification.
⎟⎠
⎞⎜⎝
⎛= ∑ )(sign)( xhxHi
iiα
![Page 23: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/23.jpg)
Using Local Intensity Patterns
• Create a set of weak classifiers that use a small image patch to classify each derivative
• The classification of a derivative:
Ip F> Tabs
![Page 24: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/24.jpg)
AdaBoost Initial uniform weight on training examples
weak classifier 1
(Freund & Shapire ’95)
⎟⎠
⎞⎜⎝
⎛= ∑
ttt xhxf )()( αθ
weak classifier 2
Incorrect classificationsre-weighted more heavily
weak classifier 3
Final classifier is weighted combination of weak classifiers
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=t
tt error
error1
log5.0α
∑ −−
−−=
i
xhyit
xhyiti
t itti
itti
eweww )(
1
)(1
α
α
Viola and Jones, Robust object detection using a boosted cascade of simple features, CVPR 2001
![Page 25: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/25.jpg)
Use Newton’s method to reduce classification cost over training set
∑ −=i
xfy iieJ )(Classification cost
Treat hm as a perturbation, and expand loss J to second order in hm
classifier with perturbation
squared error
reweighting
cost function
![Page 26: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/26.jpg)
Adaboost demo…
![Page 27: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/27.jpg)
Learning the Classifiers
• The weak classifiers, hi(x), and the weights α are chosen using the AdaBoost algorithm (see www.boosting.org for introduction).
• Train on synthetic images.• Assume the light direction is from the right.
• Filters for the candidate weak classifiers—cascade two out of these 4 categories:– Multiple orientations of 1st derivative of Gaussian filters– Multiple orientations of 2nd derivative of Gaussian filters– Several widths of Gaussian filters– impulse
![Page 28: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/28.jpg)
Classifiers Chosen
• These are the filters chosen for classifying vertical derivatives when the illumination comes from the top of the image.
• Each filter corresponds to one hi(x)
![Page 29: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/29.jpg)
Characterizing the learned classifiers
• Learned rules for all (but classifier 9) are: if rectified filter response is above a threshold, vote for reflectance.
• Yes, contrast and scale are all folded into that. We perform anoverall contrast normalization on all images.
• Classifier 1 (the best performing single filter to apply) is an empirical justification for Retinex algorithm: treat small derivative values as shading.
• The other classifiers look for image structure oriented perpendicular to lighting direction as evidence for reflectance change.
![Page 30: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/30.jpg)
Results Using Only Form Information
Shading Image Reflectance ImageInput Image
![Page 31: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/31.jpg)
Using Both Color and Form Information
Results only using chromaticity.
Shading ReflectanceInput image
![Page 32: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/32.jpg)
Some Areas of the Image Are Locally Ambiguous
Input
Shading Reflectance
Is the change here better explained as
?or
![Page 33: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/33.jpg)
Propagating Information• Can disambiguate areas by propagating
information from reliable areas of the image into ambiguous areas of the image
![Page 34: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/34.jpg)
Markov Random Fields
• Allows rich probabilistic models for images.
• But built in a local, modular way. Learn local relationships, get global effects out.
![Page 35: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/35.jpg)
Network joint probability
scene Scene-scenecompatibility
functionneighboringscene nodes
image
local observations
Image-scenecompatibility
function
∏∏ ΦΨ=i
iiji
ji yxxxZ
yxP ),(),(1),(,
![Page 36: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/36.jpg)
Inference in MRF’s
• Inference in MRF’s. (given observations, how infer the hidden states?)– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts
See www.ai.mit.edu/people/wtf/learningvision for a tutorial on learning and vision.
![Page 37: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/37.jpg)
Derivation of belief propagationy1
),( 11 yxΦ
),( 21 xxΨ
),( 22 yxΦ
),( 32 xxΨ
),( 33 yxΦ
x1
y2
x2
y3
x3
),,,,,(sumsummean 3213211321
yyyxxxPxxxxMMSE =
![Page 38: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/38.jpg)
The posterior factorizes
),(),(sum
),(),(sum
),(mean),(),(),(),(
),(sumsummean
),,,,,(sumsummean
3233
2122
111
3233
2122
111
3213211
3
2
1
321
321
xxyx
xxyx
yxxxxyxxxyx
yxx
yyyxxxPx
x
x
xMMSE
xxxMMSE
xxxMMSE
ΨΦ
ΨΦ
Φ=ΨΦΨΦ
Φ=
=
y1
),( 11 yxΦ
),( 21 xxΨ
),( 22 yxΦ
),( 32 xxΨ
),( 33 yxΦ
x1
y2
x2
y3
x3
![Page 39: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/39.jpg)
Propagation rules
y1
),( 11 yxΦ
),( 21 xxΨ
),( 22 yxΦ
),( 32 xxΨ
),( 33 yxΦ
x1
y2
x2
y3
x3),(),(sum
),(),(sum
),(mean),(),(),(),(
),(sumsummean
),,,,,(sumsummean
3233
2122
111
3233
2122
111
3213211
3
2
1
321
321
xxyx
xxyx
yxxxxyxxxyx
yxx
yyyxxxPx
x
x
xMMSE
xxxMMSE
xxxMMSE
ΨΦ
ΨΦ
Φ=ΨΦΨΦ
Φ=
=
![Page 40: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/40.jpg)
Propagation rules
y1
),( 11 yxΦ
),( 21 xxΨ
),( 22 yxΦ
),( 32 xxΨ
),( 33 yxΦ
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
ΨΦ
ΨΦ
Φ=
)( ),( ),(sum)( 23222211
21
2
xMyxxxxMx
ΦΨ=
![Page 41: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/41.jpg)
Belief, and message updates
∏∈
=)(
)( )(jNk
jkjjj xMxbj
∏∑∈
=ijNk
jkjji
xi
ji xMxxxM
j \)(ij )(),( )( ψ
= jii
![Page 42: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/42.jpg)
Optimal solution in a chain or tree:Belief Propagation
• “Do the right thing” Bayesian algorithm.• For Gaussian random variables over time:
Kalman filter.• For hidden Markov models:
forward/backward algorithm (and MAP variant is Viterbi).
![Page 43: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/43.jpg)
No factorization with loops!
y1
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
ΨΦ
ΨΦ
Φ=
31 ),( xxΨ
![Page 44: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/44.jpg)
Justification for running belief propagation in networks with loops
• Experimental results:– Error-correcting codes
– Vision applications
• Theoretical results:– For Gaussian processes, means are correct.
– Large neighborhood local maximum for MAP.
– Equivalent to Bethe approx. in statistical physics.
– Tree-weighted reparameterization
Weiss and Freeman, 2000
Yedidia, Freeman, and Weiss, 2000
Freeman and Pasztor, 1999;Frey, 2000
Kschischang and Frey, 1998;McEliece et al., 1998
Weiss and Freeman, 1999
Wainwright, Willsky, Jaakkola, 2001
![Page 45: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/45.jpg)
Region marginal probabilities
)()(),( ),(
)()( )(
\)(\)(
)(
∏∏
∏
∈∈
∈
Ψ=
Φ=
ijNkj
kj
jiNki
kijijiij
iNki
kiiii
xMxMxxkxxb
xMxkxb
i
ji
![Page 46: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/46.jpg)
Belief propagation equationsBelief propagation equations come from the
marginalization constraints.
jii
j= ii
∏∑∈
=ijNk
jkjji
xi
ji xMxxxM
j \)(ij )(),( )( ψ
![Page 47: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/47.jpg)
Results from Bethe free energy analysis
• Fixed point of belief propagation equations iff. Betheapproximation stationary point.
• Belief propagation always has a fixed point.• Connection with variational methods for inference: both
minimize approximations to Free Energy,– variational: usually use primal variables.– belief propagation: fixed pt. equs. for dual variables.
• Kikuchi approximations lead to more accurate belief propagation algorithms.
• Other Bethe free energy minimization algorithms—Yuille, Welling, etc.
![Page 48: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/48.jpg)
Kikuchi message-update rulesGroups of nodes send messages to other groups of nodes.
Typical choice for Kikuchi cluster.
i j i j=i ji
=lk
Update formessages
Update formessages
![Page 49: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/49.jpg)
Generalized belief propagationMarginal probabilities for nodes in one row
of a 10x10 spin glass
![Page 50: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/50.jpg)
References on BP and GBP
• J. Pearl, 1985– classic
• Y. Weiss, NIPS 1998– Inspires application of BP to vision
• W. Freeman et al learning low-level vision, IJCV 1999– Applications in super-resolution, motion, shading/paint
discrimination• H. Shum et al, ECCV 2002
– Application to stereo• M. Wainwright, T. Jaakkola, A. Willsky
– Reparameterization version• J. Yedidia, AAAI 2000
– The clearest place to read about BP and GBP.
![Page 51: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/51.jpg)
Propagating Information• Extend probability model to consider relationship
between neighboring derivatives
•β controls how necessary it is for two nodes to have the same label
• Use Generalized Belief Propagation to infer labels. (Yedidia et al. 2000)
⎥⎦
⎤⎢⎣
⎡−
−=
ββββ
ψ1
1),( jxxi
![Page 52: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/52.jpg)
Propagating Information• Extend probability model to consider relationship
between neighboring derivatives
•β controls how necessary it is for two nodes to have the same label
• Use Generalized Belief Propagation to infer labels. (Yedidia et al. 2000)
⎥⎦
⎤⎢⎣
⎡−
−=
ββββ
ψ1
1),( jxxi
Classification of a derivative
![Page 53: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/53.jpg)
Setting Compatibilities• All compatibilities have form
• Assume derivatives along image contours should have the same label
• Set β close to 1 when the derivatives are along a contour
• Set β to 0.5 if no contour is present
• β is computed from a linear function of the image gradient’s magnitude and orientation
0.5 1.0
β=
⎥⎦
⎤⎢⎣
⎡−
−=
ββββ
ψ1
1),( jxxi
![Page 54: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/54.jpg)
Improvements Using Propagation
Reflectance ImageWith Propagation
Input Image Reflectance ImageWithout Propagation
![Page 55: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/55.jpg)
More results…
![Page 56: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/56.jpg)
J. J. Gibson, 1968
![Page 57: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/57.jpg)
Gibson image
original
shading
reflectance
![Page 58: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/58.jpg)
Clothing catalog image
Shading ReflectanceOriginal(from LL Bean catalog)
![Page 59: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/59.jpg)
Sign at train crossing
![Page 60: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/60.jpg)
Separated images
original shading reflectanceNote: color cue omitted for
this processing
![Page 61: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/61.jpg)
![Page 62: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/62.jpg)
![Page 63: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/63.jpg)
Finally, returning to our explanatory example…
input Ideal shading image Ideal paint image
Algorithm output.Note: occluding edges labeled as reflectance.
![Page 64: Marshall F. Tappen William T. Freeman Edward H. Adelson ......Learning to separate shading from paint Marshall F. Tappen1 William T. Freeman1 Edward H. Adelson1,2 1MIT Computer Science](https://reader033.vdocuments.us/reader033/viewer/2022052815/60a6ab7b6ea72d345a50e9ac/html5/thumbnails/64.jpg)
Summary
• Sought an algorithm to separate shading and reflectance image components.
• Achieved good results on real images. • Classify local derivatives
– Learn classifiers for derivatives based on local evidence, both color and form.
• Propagate local evidence to improve classifications.
For manuscripts, see www.ai.mit.edu/people/wtf/