Intrinsic Shape Context Descriptors for Deformable Shapes
Iasonas Kokkinos
∗
Center for Visual Computing
Ecole Centrale Paris/INRIA Saclay
France
Michael M. Bronstein
†
Faculty of Informatics
Universit
`
a della Svizzera Italiana
Lugano, Switzerland
Roee Litman, Alex M. Bronstein
‡
School of Electrical Engineering
Tel Aviv University
Israel
Abstract
In this work, we present intrinsic shape context (ISC)
descriptors for 3D shapes. We generalize to surfaces the
polar sampling of the image domain used in shape con-
texts: for this purpose, we chart the surface by shooting
geodesic outwards from the point being analyzed; ‘angle’
is treated as tantamount to geodesic shooting direction,
and radius as geodesic distance. To deal with orientation
ambiguity, we exploit properties of the Fourier transform.
Our charting method is intrinsic, i.e., invariant to isometric
shape transformations. The resulting descriptor is a meta-
descriptor that can be applied to any photometric or geo-
metric property field defined on the shape, in particular, we
can leverage recent developments in intrinsic shape analy-
sis and construct ISC based on state-of-the-art dense shape
descriptors such as heat kernel signatures. Our experiments
demonstrate a notable improvement in shape matching on
standard benchmarks.
1. Introduction
Over the past decade, feature-based methods have gained
popularity in the computer vision and pattern recognition
communities with the introduction of the scale invariant fea-
ture transform (SIFT) [28] and similar algorithms [2]. The
ability of these methods to demonstrate sufficiently good
performance in many settings, including object recognition
and image retrieval and the public availability of the code
made SIFT-like approaches a first choice and a de facto
standard in a variety of image analysis tasks.
A significant milestone in the construction of this arsenal
of tools was the shape context (SC) descriptor introduced in
[3]. The original shape context approach was developed for
planar shapes, and is based on the observation that a set of
vectors connecting a point on the shape to the rest of the
shape points constitutes a rich and discriminative descrip-
tion of the shape. In order to make this description manage-
∗
Supported by ANR-10-JCJC -0205,
†
HP2C,
‡
ISFandGIFgrants.
Figure 1. Our Intrinsic Shape Context (ISC, right) descriptor is
constructed as a histogram of some field, defined on an intrinsic lo-
cal polar coordinate system (left, shown at three different points).
able, the authors proposed to compute its distribution over a
polar or log-polar system of coordinates, binning together
vectors with similar length and orientation. In this way,
each point on a shape is represented by a l ow-dimensional
descriptor that aggregates information from distant loca-
tions. The original technique was subsequently extended
to images [5, 31] where a point is described by the spatial
distribution of some vector field containing appearance in-
formation (e.g., color). This has been observed [31] to boost
matching and retrieval performance when compared to us-
ing descriptors based on solely local values of the field.
Feature-based approaches have also been considered for
1
3D shapes and curved surfaces. Early research focused
mainly on invariance under global Euclidean transforma-
tions (rigid motion). Classical works in this category in-
clude the integral volume descriptors [29, 17], spin images
[21] and multiscale local features [32] just to mention a few
out of many.
In the past decade, significant effort has been invested in
extending the invariance properties to non-rigid deforma-
tions. Some of the classical rigid descriptors were extended
to the non-rigid case by replacing the Euclidean metric with
its geodesic counterpart [20, 14]. Also, the use of conformal
factors has been proposed [27]. Being intrinsic properties of
a surface, both are independent of the way the surface is em-
bedded into the ambient Euclidean space and depend only
on its metric structure. This makes such descriptors invari-
ant to inelastic bending transformations. However, geodesic
distances suffer from strong sensitivity to topological noise,
while conformal factors, being a local quantity, are influ-
enced by geometric noise. Both types of noise, virtually
inevitable in real applications, limit the usefulness of such
descriptors.
Recently, a family of intrinsic geometric properties
broadly known as diffusion geometry has become growingly
popular. The studies of diffusion geometry are based on the
theoretical works by Berard et al. [4] and later by Coif-
man and Lafon [13] who suggested to use the eigenvalues
and eigenvectors of the Laplace-Beltrami operator associ-
ated with the shape to construct invariant metrics known as
diffusion distances. These distances as well as other dif-
fusion geometric constructs [33, 10] have been shown s ig-
nificantly more robust compared to their geodesic counter-
parts. Diffusion geometry offers an intuitive interpretation
of many shape properties in terms of spatial frequencies and
allows to use standard harmonic analysis tools [26].
In their influential paper Sun et al. [37] (and indepen-
dently, Gebal et al. [16]) introduced the heat kernel signa-
ture (HKS), based on the fundamental solutions of the heat
equation (heat kernels). A scale-invariant version of HKS
(SIHKS) was developed in [11], and a photometric HKS
in [25]. In [1], another physically-inspired descriptor, the
wave kernel signature (WKS) was proposed as a solution
to the excessive sensitivity of the HKS to low-frequency
information. As of today, these descriptors achieve state-
of-the-art performance in many deformable shape analysis
tasks such as shape retrieval [7] and 3D medical data anal-
ysis [12 ].
While being very successful in image analysis, feature
based descriptors have so far achieved a more modest suc-
cess in the analysis of surfaces. One of the difficulties in
generalizing classical 2D techniques to non-Euclidean sur-
faces stems from the fact that unlike images, surfaces lack
a global system of coordinates and can be associated with
only a local vector structure (tangent plane). Such a system
of coordinates holds only locally; trying to use it globally
like done in spin images [21] makes the descriptor sensitive
to shape deformations. Another difficulty is the fact that
geometric information is usually much poorer in features,
making it difficult to extract many repeatable and discrimi-
native features as in images. This calls for methods that can
operate everywhere on a surface, without relying on some
special treatment, e.g. to determine local orientation.
Recently, there have been attempts to adapt popular im-
age feature detectors and descriptors such as the Harris cor-
ner detector [36], difference of Gaussians and histogram of
gradients [38] to 3D shapes. In particular, the latter work
develops the gradient of a texture function plotted on a tri-
angular mesh to create SIFT-like 3D descriptors. While de-
pendent on triangulation and thus not purely intrinsic, this
line of works in one of the inspirations of our paper.
One of the reasons for the success of the shape context
descriptor is its “glocal” nature. We believe that contextual
information can greatly improve the performance of exist-
ing surface descriptors. In this paper, we propose a gener-
alization of the shape context approach to curved surfaces.
As our field function we can use texture (appearance) infor-
mation, or an intrinsic geometric signature such as HKS or
SIHKS. Our goal, as in 2D SC, is to aggregate this informa-
tion around the neighborhood of a point. The main differ-
ence is that we do it in an intrinsic manner that respects the
surface geometry, and therefore call our descriptor intrinsic
shape context (ISC).
3D Shape contexts were explored earlier in [24, 15];
but these works are not intrinsic, i.e. surface deforma-
tions affect the descriptors. An earlier work that explored
the exploitation of intrinsic geometry was [35], but there
the authors trivially deal with radial variation, by averaging
over orientations; this is strictly subsumed by our approach,
which allows us to retain and exploit the information con-
tained in the radial variation around a point.
The contribution of this paper is three-fold. First, we de-
velop a proper generalization of shape contexts to surfaces
and show several ways to construct them numerically. Sec-
ond, we show how to overcome the orientational ambiguity
that unavoidably arises in the construction of local coordi-
nate systems. Finally, we experimentally demonstrate that
the introduction of spatial context significantly improves the
discriminative power of the descriptor in matching and re-
trieval.
2. Shape Context Descriptors
The SC descriptor, originally developed for 2D shape
analysis [34] and then extended to 2D image analysis in
[5, 31], captures the values of a field 𝐼(𝑥),𝑥∈ ℝ
2
around a
point 𝑥
𝑖
in a low-dimensional descriptor 𝒮(𝑥
𝑖
) ∈ ℝ
𝑁
𝜌
×𝑁
𝜃
.
SC constructs a polar grid with 𝑁
𝜃
angular and 𝑁
𝜌
radial
bins around 𝑥
𝑖
and forms the descriptor by averaging 𝐼 over
each bin:
𝒮
𝜌,𝜃
(𝑥
𝑖
)=
ℝ
2
𝜋
𝜌,𝜃
(𝑥)𝐼(𝑥)𝑑𝑥
ℝ
2
𝜋
𝜌,𝜃
(𝑥)𝑑𝑥
, (1)
where 𝜌 ∈{𝜌
1
,...,𝜌
𝑁
𝜌
} and 𝜃 ∈{𝜃
1
,...,𝜃
𝑁
𝜃
} are cen-
ters of radial and angular bins, respectively, and 𝜋
𝜌,𝜃
=
𝜋
𝜌
𝜋
𝜃
denotes the membership function of angular bin 𝜃 and
radial bin 𝜌,
𝜋
𝜌
(𝑥)=
1 ∥𝑥 − 𝑥
𝑖
∥
2
∈ 𝑅
𝜌
,
0else,
(2)
𝜋
𝜃
(𝑥)=
1 ∠(𝑥 − 𝑥
𝑖
) ∈ 𝑅
𝜃
0else.
(3)
Here, ∥𝑥 − 𝑥
𝑖
∥
2
is the radial and ∠(𝑥 − 𝑥
𝑖
) the angular
component of the polar coordinates system around 𝑥
𝑖
, and
𝑅
𝜌
,𝑅
𝜃
denote the supports of the radial and angular bins
centered around 𝜌, 𝜃, respectively. Thus, 𝒮
𝜌,𝜃
is the average
of 𝐼(𝑥) over the bin 𝜌, 𝜃.
The hard binning in 2–3 can be replaced by soft binning
using a different membership function that assigns to each
point the probability to fall in a radial or angular bin.
3. Intrinsic Shape Context Descriptors
The main contribution of our work is the generalization
of the shape context (SC) descriptors to surfaces. For this
purpose, we adapt the polar (or log-polar) image sampling
scheme used by SC to work with two-dimensional mani-
folds (surfaces). Our treatment is intrinsic and thereby in-
variant to isometric surface deformations and embedding of
the s urface in 3D space.
The construction of the polar bins of the SC descriptor
in Eq.s 2–3 is valid only for images and cannot be applied
straightforwardly to surfaces, for two reasons. First, s uch
a construction assumes planar (zero-curvature) geometry,
while 3D objects are generally curved. Second, images
have a global unambiguous system of coordinates (allow-
ing to define the angular coordinate ∠(𝑥 − 𝑥
𝑖
)) which sur-
faces usually do not have. Consequently, generalizing SC
descriptors to surfaces involves three subproblems: chart-
ing the surface around a point, gathering statistics within
the bins of the chart, and eliminating orientation ambiguity.
We lay out our approach to these problems below.
3.1. Surface Charting
In order to build a shape context descriptor around a sur-
face point, we need to establish a local polar grid. The main
challenge here comes from the fact that the grid has to be
intrinsic, since (as opposed to the case of images) surfaces
have generally non-trivial curvature.
Let us be given a connected surface repre-
sented as a triangular mesh (𝑋, 𝐸 , 𝑇) with 𝑛
Figure 2. Three different possibilities of creating the intrinsic polar
grid: using local mapping into the plane e.g. with MDS (top left),
shooting geodesic directions from vertex (top right) and shooting
directions from a geodesic circle (bottom). The approach adopted
in this paper is the second one (top right).
vertices 𝑋 = {𝑥
1
,...,𝑥
𝑛
}, 𝑁
𝐸
edges 𝐸 =
{(𝑥
𝑖
1
,𝑥
𝑗
1
),...,(𝑥
𝑖
𝑁
𝐸
,𝑥
𝑗
𝑁
𝐸
)} s.t. (𝑥
𝑖
,𝑥
𝑗
) ∈
𝐸 iff (𝑥
𝑗
,𝑥
𝑖
) ∈ 𝐸, and 𝑁
𝑇
triangular faces 𝑇 =
{(𝑥
𝑖
1
,𝑥
𝑗
1
,𝑥
𝑘
1
),...,(𝑥
𝑖
𝑁
𝑇
,𝑥
𝑗
𝑁
𝑇
,𝑥
𝑘
𝑁
𝑇
)} s.t. (𝑥
𝑖
,𝑥
𝑗
),
(𝑥
𝑗
,𝑥
𝑘
), (𝑥
𝑖
,𝑥
𝑘
) ∈ 𝐸. The mesh can be considered as a
piece-wise planar approximation of the underlying smooth
surface.
Furthermore, we assume that the mesh is manifold (pos-
sibly with boundary), i.e., each edge is shared by at most
two triangles. Edges belonging to one triangle are bound-
ary edges, and corresponding vertices are boundary points.
The surface around boundary points is topologically equiv-
alent to a half-plane; around interior points, the surface is
locally equivalent to a plane.
Finally, we denote by 𝑑
𝑋
: 𝑋 × 𝑋 → ℝ
+
the geodesic
distance function, measuring the length of the shortest path
on the mesh (not necessarily along the edges) between any
pair of vertices.
Our goal is to create an intrinsic polar (or log-polar) grid
around a vertex 𝑥
𝑖
. The smallest 1-ring neighborhood of 𝑥
𝑖
is the set of all directly connected vertices, i.e. 𝑁
1
(𝑥
𝑖
)=
{𝑥
𝑗
:(𝑥
𝑖
,𝑥
𝑗
) ∈ 𝐸}. The creation of the grid can be done
in three different ways.
Local MDS. One way is to find a planar representa-
tion 𝜉 : 𝐵
𝑟
(𝑥
𝑖
) → ℝ
2
of a patch 𝐵
𝑟
(𝑥
𝑖
)={𝑥
𝑗
∈
𝑋 : 𝑑
𝑋
(𝑥
𝑖
,𝑥
𝑗
) ≤ 𝑟} around 𝑥
𝑖
, construct the grid on
𝜉(𝐵
𝑟
(𝑥
𝑖
)) ⊂ ℝ
2
, and then map the grid back to the surface
(Figure 2, top left). Such mappings can be established using
multidimensional scaling (MDS) [6]orconformal mapping
[19, 18, 39]. The former tries to create an isometric map-
ping from the patch to the plane by finding a planar config-
uration of points whose pairwise Euclidean distance are as
close as possible to the pairwise geodesic distances between
the surface points. However, neither MDS nor conformal
map allows to precisely map the non-Euclidean geometry
x
i
1
x
i
12
x
i
2
x
i
k
k1
'
12
'
12
x
i
1
x
i
x
i
2
x
i
k
12
Figure 3. Creation of intrinsic angular chart at interior (top row)
and boundary (middle row) point 𝑥
𝑖
. Because of surface curvature
at 𝑥
𝑖
, the sum of angles Σ 𝛼 = 𝛼
12
+ 𝛼
23
+ ...+ 𝛼
𝑘−1,𝑘
+ 𝛼
𝑘1
in
1-ring of 𝑥
𝑖
is not necessarily equal to 2𝜋. The partition is done by
mapping the 1-ring into the plane (half plane in case of a boundary
point) with angles 𝛼
′
𝑚𝑙
= 𝛼
𝑚𝑙
/Σ𝛼, dividing the plane into equal
angular segments, and mapping them back onto the 1-ring.
of the surface into the plane, thus resulting in angular or
radial distortions.
Outward ray shooting. Another approach, adopted in
this paper, is to construct the grid intrinsically (Figure 2,
top right), in a procedure bearing resemblance to the fast
marching method (FMM) used for geodesic distance com-
putation [22]. In this approach, we chart the surface locally
by shooting geodesics outwards from the vertex 𝑥
𝑖
. These
directions provide us with the counterpart of rays in a log-
polar mapping, i.e., they are surface loci with constant in-
trinsic angular coordinate. Along the same lines, we recover
the counterpart of circles in a log-polar mapping by comput-
ing geodesic distances between the point 𝑥
𝑖
and all surface
points 𝑥
𝑗
, and then recovering equidistant points (𝑟-level
sets of the function 𝑑
𝑋
(𝑥
𝑖
,𝑥)=𝑟).
The initial directions are established by partitioning the
1-ring of an interior vertex 𝑥
𝑖
into segments of equal angle
(Figure 3, top). This partition is performed as follows: we
map the 1-ring triangles (Figure 3, top left) into the plane by
partitioning the plane into angular segments (Figure 3,top
right) whose angle rations are equal to the ratios of the an-
gles in 1-ring triangles. Since the system of coordinates is
local, there is a rotation ambiguity: one of the edges is cho-
sen arbitrarily to align with the plane horizontal or vertical
axis. The plane is divided into equiangular segments. The
boundaries of these segments (denoted by red lines in Fig-
ure 3) are mapped back to the 1-ring t riangles. In the limit
this is equivalent to subdividing the tangent plane: flatten-
ing applies a common scaling to all surface triangles when
mapping to the plane, so angle ratios on the plane will be
preserved on the surface. For boundary vertices, the proce-
dure is very similar, with the exception that the mapping is
performed into a half-plane (Figure 3, bottom).
The propagation of the directions outwards from the 1-
ring is done using the standard unfolding procedure [22, 8],
which is known to be numerically consistent (refining the
mesh will converge to the geodesic, for the s ame reason that
fast marching converges). The main idea of unfolding is to
create a poly-linear path by propagating a direction across
adjacent triangles, as depicted in Figure 4.
Inward ray shooting. Finally, it is possible t o create the
angular partition by splitting a level curve of the geodesic
distance function 𝑑
𝑋
(𝑥
𝑖
,𝑥)=𝑟 into segments of equal
length, and propagate directions from these points along the
gradient ∇
𝑋
𝑑
𝑋
(𝑥
𝑖
,𝑥) towards 𝑥
𝑖
(Figure 2, bottom).
Figure 4. Propagation of direction outwards from 1-ring using the
unfolding procedure. The green triangle adjacent to the 1-ring
triangle is brought into the same plane as the one-ring triangle
(shown in dashed, top right). The direction (thick red line) is con-
tinued until it hits the triangle edge. Then, the next adjacent trian-
gle (blue) is brought into the same plane (bottom left). The pro-
cedure continues until the length of the resulting polyline reaches
some threshold.
3.2. Surface-based Statistics
Constructing a shape context descriptor requires averag-
ing the values of a field over radial and angular bins. We do
it similarly t o the 2D shape context, replacing all the defi-
nitions in Eq. 1 by their intrinsic equivalents. The intrinsic
shape context descriptor 𝒮
𝜌,𝜃
(𝑥
𝑖
) at point 𝑥
𝑖
is given by
𝒮
𝜌,𝜃
(𝑥
𝑖
)=
𝑋
𝜋
𝜌,𝜃
(𝑥)𝐼(𝑥)𝑑𝜇
𝑋
(𝑥)
𝑋
𝜋
𝜌,𝜃
(𝑥)𝑑𝜇
𝑋
(𝑥)
, (4)
where
𝑑𝜇
𝑋
(𝑥)=
1
3
𝑥
𝑖
,𝑥
𝑗
∈𝑁
1
(𝑥)
(𝑥
𝑖
,𝑥
𝑗
)∈𝐸
area(𝑥, 𝑥
𝑖
,𝑥
𝑗
)
is the local area element equal to one third the area of the
1-ring neighborhood of 𝑥, and 𝜋
𝜌,𝜃
= 𝜋
𝜌
𝜋
𝜃
, as previously,
denotes the membership function of intrinsic angular bin 𝜃
and intrinsic radial bin 𝜌.
We use soft membership functions, constructed as fol-
lows. Considering that we use 𝑁
𝜌
radial bins, we perform a
softmax assignment using the geodesic distance from 𝑥
𝑖
,
𝜋
𝜌
(𝑥)=
𝜇
𝜌
(𝑑
𝑋
(𝑥
𝑖
,𝑥))
𝜌
𝜇
𝜌
(𝑑
𝑋
(𝑥
𝑖
,𝑥))
, (5)
𝜇
𝜌
(𝑟)=
exp(
𝛽
𝜌
2
(𝑟 − 𝜌)
2
) 𝜌 ≤ 𝑁
𝜌
− 1
1
1+exp(−𝛾(𝑥−𝜌))
𝜌 = 𝑁
𝜌
.
(6)
The membership function 𝜇
𝜌
(𝑟) indicates the extent to
which radius 𝑟 belongs to geodesic ring centered around
𝜌.Ifweuse𝑁
𝜌
rings, the first 𝑁
𝜌
− 1 functions are Gaus-
sians, centered around 𝜌, while the last one is a sigmoidal,
capturing the ‘tail’ of large radii. We set the parameters
𝛽 =1/4,𝛾 =1so as to give a smooth transition among
radii.
For the angular membership function, we denote by
𝑑
𝑋,𝜃
(𝑥)=
𝑑
𝑋
(𝑣(𝑥
𝑖
,𝜃),𝑥)
𝑑
𝑋
(𝑥
𝑖
,𝑥)
the normalized point-to-set geodesic distance between a
point 𝑥 and a polyline 𝑣(𝑥
𝑖
,𝜃) shot from 𝑥
𝑖
in the direc-
tion 𝜃 (found as described in Section 3.1). The normaliza-
tion guarantees that the softness of the assignment is not
affected by the distance from 𝑥
𝑖
. The angular membership
function is given by
𝜋
𝜃
(𝑥)=
exp(−𝛼𝑑
𝑋,𝜃
(𝑥))
𝜃
exp(−𝛼𝑑
𝑋,𝜃
(𝑥))
. (7)
This softmax operation uses the distance of 𝑥 to all can-
didate directions to estimate the ‘probability’ t hat 𝑥
𝑖
be-
longs to the angular bin 𝜃. Due to normalization, we have
𝜋
𝜃
(𝑥) ∈ [0, 1] and
𝜃
𝜋
𝜃
(𝑥)=1. The parameter 𝛼 deter-
mines the hardness of the assignment and is set 𝛼 =10.
52565 Mesh points 26299 Mesh Points
5 10 15 20 25 30 35
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Descriptor index
52565 mesh points.
35056 mesh points.
26299 mesh points.
13156 mesh points.
Figure 5. Effect of surface decimation on our descriptor: the color
encodes the point, while the linestyle encodes the surface reso-
lution at which the descriptor was computed. Our descriptor is
largely invariant to decimation and remains discriminative.
3.3. Eliminating Orientation Ambiguity
Since our descriptor is intrinsic, it cannot rely on a
global system of coordinates and constructs a local polar
coordinate system around each vertex. This introduces ro-
tation ambiguity: the ray shooting procedure outlined in
Section 3.1 picks the starting orientation randomly, result-
ing in our ISC to be defined up to some unknown phase,
𝒮
𝜌,𝜃+𝑐 mod 2𝜋
. It is possible in theory to determine some
dominant reference orientation, but on surfaces -unlike
images- we do not have intensity gradients, or similar cues
to indicate orientations (for instance for a point on a sphere
all orientations are equally good).
Here, we opt for using the Fourier transform modulus
(FTM) technique to achieve rotation invariance. This tech-
nique has been used extensively in image registration [40],
and was introduced for scale- and rotation- invariant de-
scriptor construction in [23], and was also exploited to con-
struct scale-invariant heat kernel signatures (SIHKS) for
surfaces [11]. Here we exploit this technique to eliminate
