Whitened Expectation Propagation: Non-Lambertian Shape from Shading and Shadow

TL;DR: This work proposes a variation of EP that exploits regularities in natural scene statistics to achieve run times that are linear in both number of pixels and clique size, and uses large, non-local cliques to exploit cast shadow, which is traditionally ignored in shape from shading.

Abstract: For problems over continuous random variables, MRFs with large cliques pose a challenge in probabilistic inference. Difficulties in performing optimization efficiently have limited the probabilistic models explored in computer vision and other fields. One inference technique that handles large cliques well is Expectation Propagation. EP offers run times independent of clique size, which instead depend only on the rank, or intrinsic dimensionality, of potentials. This property would be highly advantageous in computer vision. Unfortunately, for grid-shaped models common in vision, traditional Gaussian EP requires quadratic space and cubic time in the number of pixels. Here, we propose a variation of EP that exploits regularities in natural scene statistics to achieve run times that are linear in both number of pixels and clique size. We test these methods on shape from shading, and we demonstrate strong performance not only for Lambertian surfaces, but also on arbitrary surface reflectance and lighting arrangements, which requires highly non-Gaussian potentials. Finally, we use large, non-local cliques to exploit cast shadow, which is traditionally ignored in shape from shading.

Summary

1. Introduction

• Probabilistic inference for large loopy graphical models has become an important subfield with a growing body of applications, including many in computer vision.
• These methods have resulted in significant progress for several applications.
• The principal difference between BP and Gaussian EP can thus be summarized by a trade-off in their respective approximating families: BP favors flexible non-Gaussian marginals, while Gaussian EP favors a flexible covariance structure.
• Another possible explanation is that for a grid-based graphical model with D pixels, Gaussian EP requires O(D2) space and a run time of O(D3).
• Finally, the authors use the method to efficiently perform inference over large cliques produced by cast shadows and by global spatial priors.

2. Expectation Propagation

• The family P̃ is chosen so that EP̃ [τj( x)] can be estimated easily.
• EP achieves this goal by approximating each potential function φi( x) with an exponential family distribution P̃i( xi| θ(i)).
• Regardless of the rank of each potential, the covariance matrix of the posterior S remains full-rank, and must be stored as a D×D matrix.
• For large problems with tens of thousands of variables or more, this becomes limiting.
• When the underlying graphical model is highly sparse, such as a nearest-neighbor pairwiseconnected MRFs, each iteration can be performed in time O(D1.5) [2].

3. Whitened EP

• For many problems of computer vision, both the number of variables D and the number of potentials N grow linearly with the number of pixels.
• Low-rank potentials of large clique size have a wide array of promising applications in computer vision [17, 10].
• Expectation propagation can be made more efficient by limiting the forms of covariance structure expressible by S. Let S denote the covariance matrix for natural scenes.

4. Shape from Shading

• Whitened EP permits inference over images in linear time with respect to both pixels and clique size.
• In particular, the authors are interested in whether Gaussian message approximation will be effective when the potentials φi are highly non-Gaussian.
• In recent years, several methods have been developed that solve the classical SfS problem well as long as surface reflectance R is assumed to be Lambertian [19, 17, 6, 3, 7].
• For each pixel, one potential φR(p, q|i) enforces the surface normal to be consistent with the known pixel intensity i(x, y).
• Whitened EP provides two benefits for spatial priors.

5. Conclusions

• The methods in this paper reduce the run time of EP from cubic to linear in the number of pixels for visual inference, while retaining a run time that is linear in clique size.
• The computational expense of inference for large cliques has prohibited the investigation of complex probabilistic models for vision.
• The authors hope is that whitened EP will facilitate further research in these directions.
• Results for whitened EP on SfS shows that the sacrifice in performance for this approach is small, even in problems with highly non-Gaussian potentials.
• Performance remained strong for surfaces with arbitrary reflectance and arbitrary lighting, which is a novel finding in SfS.

Figures

Figure 2. Results of whitened EP under several reflectances and lighting conditions. The left column shows example potential functions φR(p, q|i). In each case, potentials are highly non-Gaussian. The potential φR(p, q|i) differs at each pixel depending on intensity; here the chosen intensity is given by the blue dot on each sphere. For each reflectance, inferred surfaces are shown for benchmark SfS images. Mean-squared image error is reported for each case.

Figure 1. Results of the Whitened EP algorithm for Lambertian SfS, in comparison with other methods. Subfigure c) shows the results of linear constraint node BP [17]. Subfigure d) shows the results of EP using a full covariance matrix. Each of these methods is able to satisfy the Lambertian constraint accurately. Whitened EP is able to perform comparably in a substantially less time than other methods and with greater flexibility towards dense factor graphs or large cliques. Subfigure e) shows the results of diagonal EP without whitening.