A Symmetry Prior for Convex Variational
3D Reconstruction
Pablo Speciale
1(
B
)
, Martin R. Oswald
1
, Andrea Cohen
1
, and Marc Pollefeys
1,2
1
ETH Z¨urich, Zurich, Switzerland
pablo.speciale@inf.ethz.ch
2
Microsoft, Redmond, USA
Abstract. We propose a novel prior for variational 3D reconstruction
that favors symmetric solutions when dealing with noisy or incomplete
data. We detect symmetries from incomplete data while explicitly han-
dling unexplored areas to allow for plausible scene completions. The
set of detected symmetries is then enforced on their respective support
domain within a variational reconstruction framework. This formulation
also handles multiple symmetries sharing the same support. The pro-
p osed approach is able to denoise and complete surface geometry and
even hallucinate large scene parts. We demonstrate in several experi-
ments the benefit of harnessing symmetries when regularizing a surface.
Keywords: Symmetry prior
· 3D reconstruction · Variational methods ·
Convex optimization
1 Introduction
One of the long-time goals of computer vision algorithms is to imitate the numer-
ous powerful abilities of th e human visual system to achieve better scene un der-
standing. Many methods have actually been inspired by the physiology of the
visual cortex of mammalian brains. One of the strongest cues that humans use
in order to infer the underlying geometry of a scene despite having access to
only a parti al view is symmetry, as shown in [
20]. Moreover, symmetry is a very
strong and useful concept because it applies to many natural and man-made
environments. Following this inspiration, we propose a metho d which leverages
symmetry information directly with in a 3D reconstruction procedure in order
to complete or denoise symmetric surface regions which have been partially
occluded or where the inpu t information has low quality. In contrast to the
majority of 3D reconstruction methods which fit minimal surfaces in order to fill
unobserved surface parts, our method favors solutions which align with symme-
tries and adhere to required smoothness properties at the same time. Similarly
to how humans extrapolate occluded areas and 3D information from just a few
view points, our method can hallucinate entire scene parts in unobserved areas,
fill small holes, or denoise observed surface geometry once a symmetry has been
detected. An example of our approach is shown in Fig.
1.
Equal contribution from P. Speciale and M.R. Oswald.
c
Springer International Publishing AG 2016
B. Leibe et al. (Eds.): ECCV 2016, Part VIII, LNCS 9912, pp. 313–328, 2016.
DOI: 10.1007/978-3-319-46484-8
19
314 P. Speciale et al.
input geometry detected symmetries symmetric reconstruction
Fig. 1. Example application of our approach. A model of a stool was scanned by a
depth camera and the result is incomplete due to occlusions. With only two detected
symmetries we can complete the 5-way symmetry of the model.
1.1 Contributions
We propose to use symmetry information as a prior in 3D reconstruction in order
to favor symmetric solutions when dealing with noisy and incomplete data. For
this purpose, we extend standard symmetry detection algorithms to be able to
exploit partially unexplored domains. Our framework naturally unifies the appli-
cations of symmetry-based surface denoising, completion and the hallucination
of unexplored surface areas. To the best of our knowledge, we present the first
method that handles multiple symmetries with a shared support region, since
the proposed algorithm comput es an approximation to a non-trivial p rojection
to equally satisfy a set of symmetries. Finally, our method extends the tool-
box of priors for many existing variational 3D reconstruction methods and we
show that a symmetry prior can achieve quality improvements with a moderate
runtime overhead.
1.2 Related Work
The works by Liu et al. [
13] and Mitra et al. [16] give a broad overview of
symmetry detection methods, although most of the work they discuss focuses
on symmetry extraction for computer graphics applications. Therefore, their
applicability is mostly shown on perfect synthetic data. In this section, we focus
on works that detect and exploit symmetries from real data for vision applica-
tions.
K¨oser et al. [
11] detect planar reflective symmetries in a single 2D image.
They demonstrate that the arising stereo problem can be solved with standard
stereo matching when the distance of the camera to the symmetry plane corre-
sponds to a reasonable baseline.
Kazhdan et al. [
7] define a reflective symmetry descriptor for 3D models
by continu ously measuring symmetry scores for all planes through the models’
center of mass. We br iefly repeat some of their theoretic results as we are going
to use them in Sect.
2.Letγ ∈ Γ be a symmetric transformation and u be a
symmetric obj ect, then u is symmetric with respect to γ if it is invariant under
the symmetric transformation, that is, γ(u)=u. Using the group properties of
the symmetry transform, Kazhdan et al. define the following symmetry distance
A Symmetry Prior for Convex Variational 3D Reconstruction 315
SD(u, γ) = min
v:γ(v)=v
u − v = u − Π
γ
(u) =
u − γ(u)
2
, (1)
in which Π
γ
(u)=
1
2
(u + γ(u)) is the orthogonal projection of object u onto the
set of objects which are symmetric with resp ect to the symmetry γ.
Based on these results, Podolak et al. [23] fo cused on planar reflective sym-
metries and generalized the descriptor to detect also non-object-centered sym-
metries. They further define geometric properties which can be used for model
alignment or classification. Both [
4,22] detect symmetries in meshes and pro-
pose a symmetric remeshing of objects for qu ality improvements of the mesh
and for more consistent mesh approximations during simplification operations.
Similarly, Mitra et al. [
15] use approximate symmetry information of objects to
allow their transformation into perfectly symmetric objects.
Cohen et al. [
2] detect symmetries in sparse point clouds by using appearance-
based 3D-3D point matching in a RANSAC-based procedure. They then exploit
these symmetries to remove dr if t from point-clouds. Although their work is done
on sparse point clouds, we present a similar approach for voxel grids in Sect.
2.
Symmetries have also been used for many applications, e.g. shape matching
and feature point matching [
6], object part segmentation, and canonical coordi-
nate frame selection [23]. Our goal is to apply these concepts into the d omain of
dense 3D reconstruction. Nevertheless, our work is not the first one leveraging
symmetry information in this domain. One of th e earliest attempts to incorpo-
rate symmetry priors into surface reconstruction methods was by Terzopoulos
et al. [
25] who use a deformable spine model to create a generalized cylinder
shape from a single image.
Application-wise, the work by Thrun and Wegbreit [
26] is closely related to
ours. They detect an entire hierarchy of different symmetry types in point cloud
data and subsequently demonstrate the completion of unexplored surface parts.
As opposed to our approach, they do not simultaneously denoise the input data
and they do not compute a water-tight surface.
In contrast, we propose to integrate knowledge about symmetries directly
into the surface reconstruction process in order to better reason about noisy or
incomplete input data which can come from image-based matching algorithms
or 3D depth sensors. Furthermore, our approach can handle any number of
symmetries and their support domains can ar bit rari ly overlap. To the best of
our knowledge no other method deals with several symmetries that share the
same domain in a way that the “symmetrized” result obeys th e group structure
of several symmetries at once.
We build our symmetry prior into the variational volumetric 3D recon-
struction framework which has been used in various settings, e.g. depth map
fusion [
29], 3D reconstruction [9,27], multi-label semantic reconstruction [5], and
spatio-temporal reconstruction [
17], with anisotropic regularization [5,10,18]or
connectivity constraints [
19]. The proposed method extends these lines of works
by a symmetry prior which can be easily combined with any of these works.
316 P. Speciale et al.
Moreover, since the 3D reconstruction is formulated as a 3D segmentation prob-
lem, the proposed prior is also directly applicable to a large set of segmentation
methods like [
24,28].
2 Symmetry Detection
In order to exploit symmetry priors for reconstruction, we first need to detect
the symmetries that best fit the data. In this paper, we focus on detecting planar
symmetries. As input we use integrated depth information, which is represented
by a truncated signed distance function (TSDF) on a volume V ⊂ R [
3]. The
TSDF assigns a positive value for voxels corresponding to free space and a neg-
ative value for occupied voxels (which are placed behind the observed depth
values). A zero value in this function denotes an unobserved (or occluded) voxel.
A surface is then implicitly defined as the transition between positive and neg-
ative values. Since we are only interested in voxels lying on the surface of the
object, we only look at the voxels for which the gradient is very strong.
Furthermore, since planar surfaces are symmetric with respect to all planes
perpendicular to them, which is n ot very informative for the global scene or
even f or a small object, we decide to look only at those voxels that exhibit
a certain degree of curvature. Thus, the goal is to find the symmetry planes
that best reflect these high-gradient and high-curvature voxels into positions
that also have a large gradient and curvature or that are otherwise unknown or
occluded. Note that [
8] detect partial symmetries in volumetric data via sparse
matching of extracted line features, which is a more sophisticated way of using
the gradient on the data. While trying to find the symmetries of a scene, we
define the symmetry support V
γ
⊆ V ⊂ R
3
as a hole-free, connected subset of
the reconstruction domain V that fits the detected symmetry, that is, we try
to include all occupied and free space which complies with the symmetry γ.
Unobserved regions will be included an treated in a way such that they perfectly
fulfill the symmetry to allow for hole filling and hallucination.
2.1 RANSAC
We apply a RANSAC-based approach by taking as input the list of high-gradient
and high-curvature voxels, which we will refer to as surface voxels, and the list
of unknown voxels. First, we randomly sample two surface voxels, which define a
unique symmetry plane that reflect one of these voxels into the other. Next, we
look through all of the surface voxels and reflect them over this plane to look for
inliers to this particular symmetry. If the reflection of a surface voxel falls into
the position of another surface voxel, then they are both considered as inliers to
this symmetry plane. However, if it falls into an unknown voxel position, then
we also consider it as an inlier since this could be a potential occluded part of a
symmetric object. We randomly sample planes from two surface voxels as many
times as stated by the RANSAC termination formula.
A Symmetry Prior for Convex Variational 3D Reconstruction 317
The plane with the most inliers is chosen as the best global symmetry plane
γ for this surface, and its inliers define the support V
γ
. Since we are interested
in potentially finding many symmetries for the same scene, RANSAC could be
applied sequentially by removing the inliers for the best symmetry and then
subsequently re-detecting the n ext best symmetry among the remaining surface
voxels. However, since there could be many symmetries with the same support,
we modify RANSAC and keep track of the set of best N solutions (where N has
to be determined a-priori). This way, we extract the N symmetries that best fit
the entire surface. We will refer to this set of symmetries as Γ .
Local Symmetry Detection. A scene can be composed of several objects with
different sizes and symmetries. However, due to the size variability, applying
RANSAC on the whole volume as described before would miss most of the sym-
metries for small obj ects and also cluster different objects with similar symmetry
planes into one approximate, noisy, symmetry plane. Therefore, we apply the
described RANSAC approach for sliding b oxes of different sizes over the entire
volume. This allows the segmentation of objects as the support f or the different
symmetries found. The symmetry planes with the bigger inlier ratios and their
support are chosen as candidates for local object symmetries. For multiple detec-
tions of the same object (parts) at different scales we gave preference to larger
support domains and rejected detections whose support is a subset of another.
2.2 Hough Transform
Alternatively to RANSAC, we also implement a method based on [23], which
resembles the Hough transform approach, in order to have additional insights in
the space of planar symmetries belonging to an object. As cost for the hough
space voting, we use the Planar-Reflective Symmetry Transform (PRST), which
is defined according to [
23] as follows
PRST(u, γ)=1−
SD
2
(u, γ)
u
2
=
1+u · γ(u)
2
. (2)
We parametrize planes in 3D by the spherical coordinates of their normals θ ∈
[0,π], φ ∈ [0,π] and the distance to the origin d ∈ [d
min
,d
max
]. After finding
the peaks in the Hough Space using a non-maximal suppression scheme, we can
obtain planar symmetries with high PRST values, as illustrated in Fig.
2.
If we consider th e special case when u values are binaries, representing an
occupancy grid, the Eq. (
2) becomes the number of inliers of the γ symmet-
ric plane, a metric also used in the RANSAC method described previously.
Therefore, the methods are essentially very similar and the decision between
one or the other depends only on technical consid erations. For example, the
runtime of Hough Transform is fixed, which is the number of iterations used
in the plane sampling; on the other hand, the number of iterations ran by
RANSAC depends on the current inlier threshold and, therefore, could possi-
bly finish sooner. Another advantage of RANSAC is its low memory footprint