Belief Propagation with Directional Statistics for Solving the Shape-from-Shading Problem

TL;DR: The Shape-from-Shading problem infers shape from reflected light, collected using a camera at a single point in space only using the Fisher-Bingham distribution to marginalise a probabilistic model.

Abstract: The Shape-from-Shading [SfS] problem infers shape from reflected light, collected using a camera at a single point in space only. Reflected light alone does not provide sufficient constraint and extra information is required; typically a smoothness assumption is made. A surface with Lambertian reflectance lit by a single infinitely distant light source is also typical. We solve this typical SfS problem using belief propagation to marginalise a probabilistic model. The key novel step is in using a directional probability distribution, the Fisher-Bingham distribution. This produces a fast and relatively simple algorithm that does an effective job of both extracting details and being robust to noise. Quantitative comparisons with past algorithms are provided using both synthetic and real data.

Summary

1 Introduction

• A known or inferred reflectance function provides the relationship between irradiance and surface orientation.
• This constrained scenario has been tackled many times since[2–7, to cite a few], and will again be the focus of this work.
• More recent methods include Worthington and Hancock[5], which iterated between smoothing a normal map and correcting it to satisfy the reflectance information; Prados et al[6], which solved the problem with viscosity solutions; and Potetz[7] which used belief propagation.
• Belief propagation estimates the marginals of a multivariate probability distribution, often represented by a graphical model.

2 Formulation

• Using previously given assumptions, of Lambertian reflectance, constant known albedo, orthographic projection, an infinitely distant light source and no interreflection the irradiance at each pixel in the input image is given by Ix,y = A(̂l · n̂x,y) (1) where Ix,y is the irradiance provided by the input image.
• The normal map can be integrated to obtain a depth map, a step with which the authors are not concerned.
• By substituting the dot product with the cosine of the angle between the two vectors you get Ix,y A = cos θx,y (2) where θ is therefore the angle of a cone around l̂ which the normal is constrained to[5].
• This leaves one degree of freedom per pixel that is not constrained by the available information.
• The authors propose a new SfS algorithm using such distributions within a belief propagation framework.

3 Belief Propagation

• Such an equation can be represented by a graphical model where each variable is a node and nodes that interact via ψ functions are linked.
• Message passing then occurs within this model, with messages passed along the links between the nodes.
• The method uses belief propagation to obtain the maximum a posteriori estimate of a pairwise Markov random field where each node represents the orientation of the surface at a pixel in the image.

5 Method

• The authors construct a graphical model, specifically a pairwise Markov random field.
• For each node the authors have an irradiance value.
• The formulation presented so far will converge to a bi-modal distribution at each node, with the modes corresponding to the concave and convex interpretations.
• Each level’s messages are initialised with the previous, lower resolution, levels messages.

6 Message Passing

• Doing this directly is not tractable, so the authors propose a novel three step procedure to solve this problem: 1. Convert the FB8 distribution into a sum of Fisher distributions.
• All three steps involve approximation, in practise this proves not to be a problem.
• To derive a Fisher-Bingham distribution from the convolved sum of Fisher distributions the authors first need the rotational component of the Bingham distribution, which they calculate with principal component analysis.
• This is irrelevant as multiplicative constants have no effect.

7 Results & Analysis

• The authors compare the presented algorithm to two others, Lee & Kuo[4] and Worthington & Hancock[5], using both synthetic and real data.
• Figure 1 gives the four synthetic inputs used, figure 2 gives the results and ground truth for just one of the four inputs.
• For the Mozart 90◦ input their approach consistently exceeds Lee & Kuo but does not do so well at getting a high percentage of spot on estimates as Worthington & Hancock.
• The presented algorithm doing poorly as the light source moves away from [0, 0, 1]T can be put down to the bias introduced to handle the concave/convex ambiguity[12].
• Looking at figure 5 Worthington & Hancock is quantitatively ahead, but looking at the actual output it is more blob than face, though some features are recognisable.

8 Conclusion

• The authors have presented a new algorithm for solving the classical shape from shading algorithm, and demonstrated its competitiveness with previously published algorithms.
• The use of belief propagation with FB8 distributions is in itself new, and a method for the convolution of a FB8 distribution by a Fisher distribution has been devised.
• The algorithm does suffer a noticeable flaw in that overcoming the convex/concave problem biases the result, making the algorithm weak in the presence of oblique lighting.
• An alternative solution to the current bias is an obvious area for future research.

Figures

Fig. 1. Synthetic inputs, derived from the set used by Zhang et al[8]. From left to right they are referred to as Vase 90◦, Vase 45◦, Mozart 90◦ and Mozart 45◦. The light source direction vector for the 90◦ images is [0, 0, 1]T , whilst for the 45◦ images it is [− √ 2, 0, √ 2]T .

Fig. 2. Results for the synthetic Mozart 90◦ input. From left to right they are Lee & Kuo[4], Worthington & Hancock[5], the presented algorithm and then finally ground truth. They represent normal maps, with x→ red, y → green and z → blue to represent the surface normal at each pixel. Red and Green are adjusted to cover the whole [−1, 1] range, blue is left covering [0, 1].

Fig. 4. Input and results for the head. From left to right they are input, Lee & Kuo[4] and Worthington & Hancock[5] on the first line and the presented algorithm and then ground truth on the second.

Fig. 5. Results for head input. See figure 3 for explanation.

Fig. 3. Synthetic results. Each grid gives results for the input named in the top left. Each row gives results for a specific algorithm. Each column gives the percentage of pixels within a given error bound, i.e. the < 1◦ column gives the percentage of pixels where the estimated surface orientation is within 1 degree of the ground truth. The percentage is only for pixels where ground truth is provided.

Full text

Citation for published version:
Fincham Haines, T & Wilson, RC 2008, 'Belief propagation with directional statistics for solving the shape-from-
shading problem', Paper presented at European Conference on Computer Vision, Marseille, France, 12/10/08 -
18/10/08 pp. 780-791.
Publication date:
2008
Document Version
Peer reviewed version
Link to publication
University of Bath
Alternative formats
If you require this document in an alternative format, please contact:
openaccess@bath.ac.uk
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners
and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Download date: 10. Aug. 2022

Belief Propagation with Directional Statistics for
solving the Shape-from-Shading problem
Tom S. F. Haines and Richard C. Wilson
The University of York,
Heslington, YO10 5DD, U.K.
Abstract. The Shape-from-Shading [SfS] problem infers shape from re-
flected light, collected using a camera at a single point in space only.
Reflected light alone does not provide sufficient constraint and extra
information is required; typically a smoothness assumption is made. A
surface with Lambertian reflectance lit by a single infinitely distant light
source is also typical.
We solve this typical SfS problem using belief propagation to marginalise
a probabilistic model. The key novel step is in using a directional prob-
ability distribution, the Fisher-Bingham distribution. This produces a
fast and relatively simple algorithm that does an effective job of both
extracting details and being robust to noise. Quantitative comparisons
with past algorithms are provided using both synthetic and real data.
1 Intro duction
The classical problem of Shape-from-Shading [SfS] uses irradiance captured by a
photo to calculate the shape of a scene. A known or inferred reflectance function
provides the relationship between irradiance and surface orientation. Surface ori-
entation may then be integrated to obtain a depth map. Horn[1] introduced this
problem with the assumptions of Lambertian reflectance, orthographic projec-
tion, constant known albedo, a smooth surface, no surface inter-reflectance and
a single infinitely distant light source in a known relation with the photo. This
constrained scenario has been tackled many times since[2–7, to cite a few], and
will again be the focus of this work.
Zhang et al.[8] surveyed the area in 1999, concluding that Lee and Kuo[4]
was the then state of the art. Lee and Kuo iteratively linearised the reflectance
map and solved the resulting linear equation using the multigrid method. More
recent methods include Worthington and Hancock[5], which iterated between
smoothing a normal map and correcting it to satisfy the reflectance informa-
tion; Prados et al[6], which solved the problem with viscosity solutions; and
Potetz[7] which used belief propagation. This last work by Potetz is particularly
relevant due to it also using belief propagation, though in all further details it
differs. Belief propagation estimates the marginals of a multivariate probability
distribution, often represented by a graphical model. Potetz makes use of two
variables per pixel, δx/δz and δy/δz, and uses various factor nodes to provide
the reflectance information, smoothness assumption and integrability constraint.

2 T. S. F. Haines and R. C. Wilson
Whilst this model can be implemented simply with discrete belief propagation
it would never converge and require a large number of labels, instead advanced
continuous methods are used.
The following three sections, 2 through to 4, cover the component details,
starting with the formulation, then belief propagation and finally directional
statistics. Section 5 brings it all together into a cohesive whole, and is followed
by section 6 which solves a specific problem. Following these sections we give
results in section 7 and conclusions in the final section.
2 Formulation
Using previously given assumptions, of Lambertian reflectance, constant known
albedo, orthographic projection, an infinitely distant light source and no inter-
reflection the irradiance at each pixel in the input image is given by
I
x,y
= A(
ˆ
l ·
ˆ
n
x,y
) (1)
where I
x,y
is the irradiance provided by the input image. A is the albedo and
ˆ
l ∈ R
3
, |
ˆ
l| = 1 is the direction to the infinitely distant light source; these are both
provided by the user.
ˆ
n
x,y
∈ R
3
, |
ˆ
n
x,y
| = 1 is the normal map to be inferred as
the algorithm’s output. The normal map can be integrated to obtain a depth
map, a step with which we are not concerned. By substituting the dot product
with the cosine of the angle between the two vectors you get
I
x,y
A
= cos θ
x,y
(2)
where θ is therefore the angle of a cone around
ˆ
l which the normal is constrained
to[5]. This leaves one degree of freedom per pixel that is not constrained by the
available information. A smoothness assumption provides the extra constraint.
Directional statistics is the field of statistics on directions, such as surface
normals. Using a directional distribution allows the representation of surface
orientation with a single variable, rather than the two used in Potetz[7] and
many others. We propose a new SfS algorithm using such distributions within
a belief propagation framework. This leads to a belief propagation formulation
not dissimilar to Gaussian belief propagation[9] in its simplicity and speed.
3 Belief Propagation
Loopy sum-product belief propagation is a message passing algorithm for marginal-
ising an equation of the form
P (x) =
Y
v∈V
ψ
v
(y
v
) (3)
where x is a set of random variables and ∀v, y
v
⊂ x. Such an equation can be
represented by a graphical model where each variable is a node and nodes that

BP with Directional Statistics for solving the SfS problem 3
interact via ψ functions are linked. In this case the random variables are di-
rections, represented by normalised vectors. Message passing then occurs within
this model, with messages passed along the links between the nodes. As the vari-
ables are directions the messages are probability distributions on directions. The
method uses belief propagation to obtain the maximum a posteriori estimate of
a pairwise Markov random field where each node represents the orientation of
the surface at a pixel in the image. The message passed from node p to node q
at iteration t is
m
t
p→q
(
ˆ
x
q
) =
Z
ˆ
x
p
ψ
pq
(
ˆ
x
p
,
ˆ
x
q
)ψ
p
(
ˆ
x
p
)
Y
u∈(N\q)
m
t−1
u→p
(
ˆ
x
p
)δ
ˆ
x
p
(4)
where ψ
pq
(
ˆ
x
p
,
ˆ
x
q
) is the compatibility between adjacent nodes, ψ
p
(
ˆ
x
p
) is the
prior on each node’s orientation and N is the 4-way neighbourhood of each
node. Once message passing has iterated sufficiently for convergence to occur
the belief at each node is
b
p
(
ˆ
x
p
) = ψ
p
(
ˆ
x
p
)
Y
u∈N
m
t−1
u→p
(
ˆ
x
p
) (5)
From b
p
(
ˆ
x
p
) the most probable direction is selected as output.
4 Directional Statistics
The Fisher distribution, using proportionality rather than a normalising con-
stant, is given by
P
F
(
ˆ
x; u) ∝ exp(u
T
ˆ
x) (6)
where
ˆ
x, u ∈ R
3
and |
ˆ
x| = 1. Similarly, the Bingham distribution may be defined
as
P
B
(
ˆ
x; A) ∝ exp(
ˆ
x
T
A
ˆ
x) (7)
where A = A
T
. By multiplying the above we get the 8 parameter Fisher-
Bingham[10] [FB
8
] distribution
P
FB
8
(
ˆ
x; u, A) ∝ exp(u
T
ˆ
x +
ˆ
x
T
A
ˆ
x) (8)
All three of these distributions have the advantage that they can be multiplied
together without introducing further variables, which is critical in a belief prop-
agation framework. We may decompose the FB
8
distribution. As A is symmetric
we may apply the eigen-decomposition to obtain A = BDB
T
, where B is or-
thogonal and D diagonal. This allows us to write
P
FB
8
(
ˆ
x; u, A) ∝ exp(v
T
ˆ
y +
ˆ
y
T
D
ˆ
y) (9)
where v = B
T
u and
ˆ
y = B
T
ˆ
x. As |
ˆ
y| = 1 we may offset D by an arbitrary
multiple of the identity matrix, this allows any given entry to be set to 0. We
can therefore consider it the case that D = Diag(α, β, 0), with α > 0 and β > 0
so that
P
FB
8
(
ˆ
x; u, A) ∝ exp(v
T
ˆ
y + α
ˆ
y
2
x
+ β
ˆ
y
2
y
) (10)

4 T. S. F. Haines and R. C. Wilson
For convenience we may represent the FB
8
distribution as
exp(u
T
ˆ
x +
ˆ
x
T
A
ˆ
x) = Ω[u, A] (11)
Using this notation multiplication is
Ω[u, A]Ω[v, B] = Ω[u + v, A + B] (12)
Various distributions may be represented by the Fisher-Bingham distribu-
tion, of particular use is the Bingham-Mardia distribution[11]
exp(−k(
ˆ
u
T
ˆ
x − cos θ)
2
) = Ω[2k cos(θ)
ˆ
u, −k
ˆ
u
ˆ
u
T
] (13)
where
ˆ
u is the direction of the axis of a cone and θ the angle of that cone.
This distribution has a small circle as its maximum, which allows the irradiance
information (Eq. 2) to be expressed as a FB
8
distribution.
5 Method
We construct a graphical model, specifically a pairwise Markov random field.
Each node of the model is a random variable that represents an unknown normal
on the surface. Belief propagation, as described in section 3, is then used to
determine the marginal distribution for each node. To define the distribution to
be marginalised two sources are used: the irradiance information (Eq. 2) and a
smoothness assumption.
We model the smoothing assumption on the premise that adjacent points on
the surface will be more likely to have a small angular difference than a large
angular difference. We can express this idea by setting
ψ
pq
(
ˆ
x
p
,
ˆ
x
q
) = exp(k(
ˆ
x
T
p
ˆ
x
q
)) (14)
where ψ
pq
(
ˆ
x
p
,
ˆ
x
q
) is from the message passing equation (Eq. 4). This is a Fisher
distribution with concentration k. Using FB
8
for the messages and dropping
equation 14 into equation 4 we have
m
t
p→q
(
ˆ
x
q
) =
Z
S
2
exp(k(
ˆ
x
T
p
ˆ
x
q
))t(
ˆ
x
p
)δ
ˆ
x
p
(15)
t(
ˆ
x
p
) = ψ
p
(
ˆ
x
p
)
Y
u∈(N\q)
m
t−1
u→p
(
ˆ
x
p
) (16)
Message passing therefore consists of two steps: calculating t(
ˆ
x
p
) by multiplying
FB
8
distributions together using equation 12, followed by convolution of the
resulting FB
8
distribution by a Fisher distribution to get m
t
p→q
(
ˆ
x
q
). The next
section documents a method for doing the convolution.
For each node we have an irradiance value. Using equations 2 and 13 we can
define a distribution
Ω[2k
I
x,y
A
ˆ
l, −k
ˆ
l
ˆ
l
T
] (17)

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "Belief propagation with directional statistics for solving the shape-from-shading problem" ?

In this paper, the authors use belief propagation to marginalise a probabilistic model and use a directional probability distribution, the Fisher-Bingham distribution, to estimate the marginals of a multivariate probability distribution. 

Q2. What have the authors stated for future works in "Belief propagation with directional statistics for solving the shape-from-shading problem" ?

An alternative solution to the current bias is an obvious area for future research. 

Q3. What is the reason why the presented algorithm does poorly?

The presented algorithm doing poorly as the light source moves away from [0, 0, 1]T can be put down to the bias introduced to handle the concave/convex ambiguity[12]. 

Q4. How long did the algorithm take to produce the head image?

For the head image the run time is over 12 hours for Lee & Kuo, 54 minutes for Worthington and Hancock and 9.5 minutes for the presented algorithm on a 2Ghz Athlon. 

Q5. What is the method used to obtain the maximum a posteriori estimate of a pairwise?

The method uses belief propagation to obtain the maximum a posteriori estimate of a pairwise Markov random field where each node represents the orientation of the surface at a pixel in the image. 

Q6. What is the definition of the distribution to be marginalised?

To define the distribution to be marginalised two sources are used: the irradiance information (Eq. 2) and a smoothness assumption. 

Q7. What is the irradiance at each pixel in the input image?

Using previously given assumptions, of Lambertian reflectance, constant known albedo, orthographic projection, an infinitely distant light source and no interreflection the irradiance at each pixel in the input image is given byIx,y = A(̂l · n̂x,y) (1)where Ix,y is the irradiance provided by the input image. 

Q8. How do the authors define a distribution to be marginalised?

Using equations 2 and 13 the authors can define a distributionΩ[2k Ix,y A l̂,−kl̂̂lT ] (17)In principle ψp(x̂p), from equation 16, can be set to this Bingham-Mardia distribution to complete the model to be marginalised.