IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 30, NO. 1, JANUARY 2000 67
Correspondence________________________________________________________________________
Isolated 3-D Object Recognition through Next View
Planning
Sumantra Dutta Roy, Santanu Chaudhury, and Subhashis Banerjee
Abstract—In many cases, a single view of an object may not contain suf-
ficient features to recognize it unambiguously. This paper presents a new
on-line recognition scheme based on next view planning for the identifica-
tion ofanisolatedthree-dimensional(3-D) objectusingsimple features. The
scheme uses a probabilistic reasoning framework for recognition and plan-
ning. Our knowledge representation scheme encodes feature based infor-
mation about objects as well as the uncertainty in the recognition process.
This is used both in the probability calculations as well as in planning the
next view. Results clearly demonstrate the effectiveness of our strategy for
a reasonably complex experimental set.
Index Terms—Active vision, reactive planning, 3-D object recognition.
I. INTRODUCTION
In this paper, we present a new on-line scheme for the recognition
of an isolated three-dimensional (3-D) object using reactive next view
planning. A hierarchical knowledge representation scheme facilitates
recognition and the planning process. The planning process utilizes
the current observation and past history for identifying a sequence of
moves to disambiguate between similar objects.
Most model-based object recognition systems consider the problem
of recognizing objects from the image of a single view [1]–[4]. How-
ever, a single view may not contain sufficient features to recognize
the object unambiguously. In fact, two objects may have all views in
common with respect to a given feature set, and may be distinguished
only through a sequence of views. Further, in recognizing 3-D objects
from a single view, recognition systems often use complex feature sets
[2]. In many cases, it may be possible to achievethe same, incurring less
error and smaller processing cost using a simpler feature set and suit-
ably planning multiple observations. A simple feature set is applicable
for a larger class of objects than a model base specific complex feature
set. Model base-specific complex features such as 3-D invariants have
been proposed only for special cases so far (e.g., [3]). The purpose of
this paper is to investigate the use of suitably planned multiple views
and two-dimensional (2-D) invariants for 3-D object recognition.
A. Relation with Other Work
With an active sensor, object recognition involves identification of a
view of an object and if necessary, planning further views. Tarabanis
et al. [5] survey the field of sensor planning for vision tasks. We can
compare various active 3-D object recognition systems on the basis of
the following four issues.
1) Nature of the Next View Planning Strategy: The system should
plan moves with maximum ability to discriminate between views
Manuscript received October 23, 1997; revised May 5, 1998.
S. Dutta Roy and S. Banerjee are with the Department of Computer Sci-
ence and Engineering, Indian Institute of Technology, NewDelhi-110 016, India
(e-mail: sumantra@ee.iitd.ernet.in; suban@cse.iitd.ernet.in).
S. Chaudhury is with the Department of Electrical Engineering, Indian Insti-
tute of Technology, New Delhi 110 016, India.
Publisher Item Identifier S 1083-4427(00)01177-2.
common to more than one object in the model base. The cost in-
curred in this processshould also be minimal. The system should,
preferably be on-line and reactive—the past and present inputs
should guide the planning mechanism at each stage.
While the scheme of Maver and Bajcsy [6] is on-line, that of
Gremban and Ikeuchi [7] is not. Due to the combinatorial nature
of the problem, an off-line approach may not always be feasible.
2) Uncertainty Handling Capability of the Hypothesis Generation
Mechanism: The occlusion-based next view planning approach
of Maver and Bajcsy [6], as well as that of Gremban and Ikeuchi
[7] are essentially deterministic. A probabilistic strategy can
make the system more robust and resistant to errors compared to
a deterministic one. Dickinson et al. [8] use Bayesian methods
to handle uncertainty, while Hutchinson and Kak [9] use the
Dempster–Shafer theory.
3) Efficient Representation of Domain Knowledge: Theknowledge
representation scheme should support an efficient mechanism
to generate hypotheses on the basis of the evidence received. It
should also play a role in optimally planning the next view.
Dickinson et al. [8] use a hierarchical representation scheme
based on volumetric primitives, which are associated with a high
feature extraction cost. Due to the non-hierarchical nature of
Hutchinson and Kak’s system [9], many redundant hypotheses
are proposed, which have to be later removed through consis-
tency checks.
4) Speed and Efficiency of Algorithms for Both Hypothesis Gen-
eration and Next View Planning: It is desirable to have algo-
rithms with low order polynomial-time complexity to generate
hypotheses accurately and fast. The next view planning strategy
acts on the basis of these hypotheses.
In Hutchinson and Kak’s system [9], although the poly-
nomial-time formulation overcomes the exponential time
complexity associated with assigning beliefs to all possible
hypotheses, their system still has the overhead of intersection
computation in creating common frames of discernment. Con-
sistency checks have to be used to remove the many redundant
hypotheses produced earlier. Though Dickinson et al. [8] use
Bayes nets for hypothesis generation, their system incurs the
overhead of tracking the region of interest through successive
frames.
The next view planning strategy that this paper presents is reactive
and on-line—the evidence obtained from each view is used in the hy-
pothesis generation and the planning process. Our probabilistic hypoth-
esis generation mechanism can handle cases of feature detection errors.
We use a hierarchical knowledge representation scheme which not only
ensures a low-order polynomial-time complexity of the hypothesis gen-
eration process, but also plays an important role in planning the next
view. The hierarchy itself enforces different constraints to prune the
set of possible hypotheses. The scheme is independent of the type of
features used, unlike that of [8]. We present results of over 100 exper-
iments with our recognition scheme on two sets of models. Extensive
experimentation shows the effectiveness of our proposed strategy of
using simple features and multiple views for recognizing complex 3-D
shapes.
The organization of the rest of the paper is as follows: Section II
presents our knowledge representation scheme. We discuss hypothesis
generation for class and object recognition in Section III. Section IV
1083-4427/00$10.00 © 2000 IEEE
68 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 30, NO. 1, JANUARY 2000
describes our algorithm for planning the next view. In Section V we
demonstrate the working of our system on two sets of objects. We sum-
marize the salient features of our scheme and discuss areas for further
work in Section VI.
II. T
HE KNOWLEDGE REPRESENTATION SCHEME
A view of a 3-D object is characterized by a set of features. With re-
spect to a particular feature set and over a particular range of viewing
angles, a view of a 3-D object is independent of the viewpoint. Koen-
derink and van Doorn [10] define aspects as topologically equivalent
classes of object appearances. Ikeuchi et al. generalize this definition:
object appearances may be grouped into equivalence classes with re-
spect to a feature set. These equivalence classes are aspects [11]. In this
context, we define the following terms:
Class A: Class (or, aspect-class) is a set of aspects, equiva-
lent with respect to a feature set.
Feature-Class: A feature-class is a set of equivalent aspects de-
fined for one particular feature.
Fig. 1 shows a simple example of an object with its associated aspects
and classes. The locus of view-directions is one-dimensional (1-D) and
we assume orthographic projection. The basis of the different classes is
the number of horizontal lines
(
h
)
and vertical lines
(
v
)
in a particular
view of the object. Thus, a class may be represented as
h
hv
i
. There
are six aspects of the object shown, belonging to three classes. In this
example, for simplicity we assume only one feature detector so that
each feature-class is also a class.
We propose a new knowledge representation scheme encoding do-
main knowledge about the object, relations between different aspects,
and the correspondence of these aspects with feature detectors. Fig. 2
illustrates an example of this scheme. We use this knowledge represen-
tation scheme both in belief updating as well as in next view planning.
Sections III and
IV discuss these topics, respectively. The representa-
tion scheme consists of two parts.
1) The Feature-Dependence Subnet: In the feature-dependence
subnet
• F represents the complete set of features
f
F
j
g
used for
characterizing views.
• A feature node
F
j
is associated with feature-classes
f
jk
.
Factors such as noise and nonadaptive thresholds can introduce
errors in the feature detection process. Let
p
jlk
represent the
probability that the feature-class present is
f
jl
, given that the de-
tector for feature
F
j
detects it to be
f
jk
. We define
p
jlk
as the
ratio of the number of times the detector for feature
F
j
interprets
feature-class
f
jl
as
f
jk
, and the number of times the feature de-
tector reports the feature-class as
f
jk
. The
F
j
node stores a table
of these values for its corresponding feature detector.
• A class node
C
i
stores its a priori probability,
P
(
C
i
)
.A
link between class
C
i
and feature-class
f
jk
indicates that
f
jk
forms a subset of features observed in
C
i
. This ac-
counts for a PART-OF relation between the two. Thus, a
class represents an
n
-vector
[
f
1
j
f
2
j
111
f
nj
]
. Since a
class cannot be independent of any feature, each class has
n
input edges corresponding to the
n
features.
2 The Class-Aspect Subnet: The class-aspect subnet encodes the
relationships between classes, aspects, and objects.
• O represents the set of all objects
f
O
i
g
• An object node
O
i
stores its probability,
P
(
O
i
)
.
• An aspect node
a
ij
stores its angular extent
ij
(in de-
grees), its probability
P
(
a
ij
)
, its parent class
C
j
, and its
neighboring aspects.
• Aspect
a
ij
has a PART-OF relationship with its parent ob-
ject
O
i
. Thus,
3
-tuple
h
O
i
;C
j
;
ik
i
represents an aspect.
Fig. 1. Aspects and classes of an object.
Fig. 2. Example of the knowledge representation scheme.
Aspect node
a
ij
has exactly one link to any object
(
O
i
)
and exactly one link to its parent class
C
j
.
III. H
YPOTHESIS GENERATION
The recognition system takes any arbitraryviewof an object as input.
Using a set of features (the feature-classes), it generates hypotheses
about the likely identity of the class. This is, in turn used for gener-
ating hypotheses about the object’s identity. The interaction of the hy-
pothesis generation part with the rest of the system is shown in Fig. 3.
Hypothesis generation consists of two steps namely, class identifica-
tion, and object identification.
A. Class Identification, Accounting for Uncertainty
Our algorithm suitably schedules feature detectors to perform prob-
abilistic class identification. In what follows, we discuss its various as-
pects. Fig. 4 presents the overall algorithm.
1) Ordering of Feature Detectors: A proper ordering of feature de-
tectors speeds up the class recognition process. At any stage, we choose
the hitherto unused feature detector for which the feature-class corre-
sponding to the most probable class has the least number of outgoing
arcs, i.e., the least out-degree. This is done in order to obtain that fea-
ture-class which has the largest discriminatory power in terms of the
number of classes it could correspond to. For example, in Fig. 2 if all
feature detectors are unused and
C
2
has the highest a priori probability,
F
3
will be tried first, followed by
F
2
and
F
1
, if required.
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 30, NO. 1, JANUARY 2000 69
Fig. 3. Flow diagram depicting the flow of information and control in our system.
Fig. 4. Class recognition algorithm.
2) Class Probability Calculations Using the Knowledge Represen-
tation Scheme: We obtain the a priori probability of class
C
i
as
P
(
C
i
)=
p
P
(
O
p
)
1
q
P
(
a
pq
j
O
p
)
:
(1)
Here, aspects
a
pq
belong to class
C
i
. Let
N
F
,
N
C
, and
N
a
denote
the number of feature-classes associated with feature detector
F
j
, the
number of classes, and the number of aspects, respectively.
P
(
a
pq
j
O
p
)
is
pq
=
360
.We can compute
P
(
C
i
)
from our knowledgerepresentation
scheme by considering each aspect node belonging to an object and
testing if it has a link to node
C
i
; this takes
O
(
N
C
+
N
a
)
time. (The
N
C
term is for the initialization of class probabilities to 0.)
Let the detector for feature
F
j
report the feature-class obtained to be
f
jk
. Given this evidence, we obtain the probability of class
C
i
from the
Bayes rule
P
(
C
i
j
f
jk
)=
P
(
C
i
)
1
P
(
f
jk
j
C
i
)
m
[
P
(
C
m
)
1
P
(
f
jk
j
C
m
)]
(2)
P
(
f
jk
j
C
i
)
is 1 for those classes which have a link from feature-class
f
jk
. It is 0 for the rest. The computation of (2) takes
O
(
N
C
)
time—this
is done for each feature-class. Hence, the computation of
P
(
f
jk
j
C
i
)
for
all feature-classes
f
jk
for feature detector
F
j
takes time
O
(
N
F
1
N
C
)
.
For an error-free situation,
P
(
C
i
j
f
jk
)
is
P
0
(
C
i
)
, the a posteriori
probability of class
C
i
. However, due to errors possible in the feature
detection process, a degree of uncertainty is associated with the evi-
dence. The value of
P
0
(
C
i
)
is, then
P
0
(
C
i
)=
l
P
(
C
i
j
f
jl
)
1
p
jlk
(3)
where
f
jl
’s are feature-classes associated with feature
F
j
. According
to our knowledge representation scheme, only one feature-class under
feature
F
j
, say
f
jr
has a link to class
C
i
. The summation reduces to one
term,
P
(
C
i
j
f
jr
)
1
p
jrk
. Thus, our knowledge representation scheme
also enable recovery from feature detection errors.
B. Object Identification
Based on the outcome of the class recognition scheme, we estimate
the object probabilities as follows. Initially, we calculate the a priori
probability of each aspect as
P
(
a
j k
)=
P
(
O
j
)
1
P
(
a
j k
j
O
j
)
:
(4)
70 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 30, NO. 1, JANUARY 2000
(a) (b)
Fig. 5. (a) The notation used (
Section IV) and (b) a case when our algorithm
is not guaranteed to succeed (Section IV-A).
If there are
N
objects in the model base, weinitialize
P
(
O
j
)
to
1
=N
before the first observation. For the first observation,
P
(
a
j k
j
Oj
p
)
is
j k
=
360
. A priori aspect probability calculations take
O
(
N
a
)
time.
For any subsequent observation, we have to account for the move-
ment in the probability calculations. For example, a particular move-
ment may preclude the occurrence of some aspects for a given class
observed. The value of
P
(
a
j k
j
O
j
)
is given by
P
(
a
j k
j
O
j
)=
j k
=
360
(5)
where
j k
(
j k
2
[0
;
j k
])
represents the angular range pos-
sible within aspect
a
j k
for the move(s) taken to reach this posi-
tion. Due to the movement made, we could have observed only
m
(0
m
r
)
aspects out of a total of
r
aspects belonging to class
C
i
.
Experiments with Model Base I
Let the class recognition phase report the observed class to be
C
i
.
Let us assume that
C
i
could have come from aspects
a
j
k
,
a
j k
,
111
;a
j k
, where all
j
1
;j
2
;
111
;j
m
are not necessarily different.
We obtain the a posteriori probability of aspect
a
j k
given this evi-
dence using the Bayes rule
P
(
a
j k
j
C
i
)=
P
(
a
j k
)
1
P
(
C
i
j
a
j k
)
m
p
=1
[
P
(
a
j k
)
1
P
(
C
i
j
a
j k
)]
(6)
P
(
C
i
j
a
j k
)
is
1
for aspects with a link to class
C
i
,0 otherwise. Finally,
we obtain the a posteriori probability
P
(
O
j
)=
l
P
(
a
j k
j
C
i
)
(7)
where aspects
a
j k
belong to class
C
i
.
If the probability of some object is above a predetermined threshold
(experimentally determined, e.g., 0.87 for Model Base I), the algorithm
reports a success, and stops. If not, it means that the view of the object
is not sufficient to identify the object unambiguously. We have to take
the next view.
In our hierarchical scheme, the link conditional probabilities (rep-
resenting relations between nodes) themselves enforce consistency
checks at each level of evidence. The feature evidence is progressively
refined as it passes through different levels in the hierarchy, leading to
simpler evidence propagation and less computational cost. This is an
advantage of our scheme over that proposed in [9].
IV. N
EXT VIEW PLANNING
The class observed in the class recognition phase could have come
from many aspects in the model base, each with its own range of po-
sitions within the aspect. Due to this ambiguity, one has to search for
Fig. 6. Partially constructed search tree.
Fig. 7. Object recognition algorithm.
the best move to discern between these competing aspects subject to
memory and processing limitations, if any. The parameters described
above characterize the state of the system. The planning process aims
to determine a move from the current step, which would uniquely iden-
tify the given object. We pose the planning problem as that of a forward
search in the state space which takes us to a state in which the aspect
list corresponding to the class observed has exactly one node. We use a
search tree for this purpose. A search tree node represents the following
information: [Fig. 5(a)] the unique class observedfor the angular move-
ment made so far, the aspects possible for this angle-class pair, and for
each aspect, the range of positions possible within it
(
s
ij
0
e
ij
)
.
s
ij
and
e
ij
denote the two positions within aspect
a
ij
where the current
viewpoint can be, as a result of the movement made thus far. Here,
s
ij
e
ij
; and
s
ij
,
e
ij
2
[0
;
ij
]
, where
ij
is the angular extent of
aspect
a
ij
. A leaf node is one which has either one aspect associated
with it or corresponds to a total angular movement of 360
or more
from the root node.
Fig. 6 shows an example of a partially constructed search tree. From
a view point, we categorize possible moves as follows.
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 30, NO. 1, JANUARY 2000 71
Fig. 8. Model Base I: The objects (from left) are
O
,
O
,
O
,
O
,
O
,
O
,
O
, and
O
, respectively.
(a)
(b)
(c)
(d)
Fig. 9. Some experiments with Model Base I: initial class
h
232
i
. The objects are
O
[(a) and (c)], and
O
[(b) and (d)], respectively. (a)
h
232
i !h
231(221)
i !h
232
i !h
221
i !h
232
i
. (b)
h
232
i !h
221
i !h
221
i !h
221
i
. (c)
h
232
i !h
232
i !h
221
i
. (d)
h
232
i !h
221
i !h
221
i !h
221
i
. The numbers above the arrows denote the number of turntable steps. A negative sign indicates a clockwise movement.
(The figure in parentheses shows an example of recovery from feature detection errors.)
Primary Move: A primary move represents a move from an aspect
by
, the minimum angle needed to move out of it.
Auxiliary Move: An auxiliary move represents a move from an as-
pect by an angle corresponding to the primary move of another com-
peting aspect.
Let
c
ij
and
a
ij
represent the minimum angles necessary to move out
of the current assumed aspect in the clockwise and counterclockwise
directions, respectively. Three cases are possible.
1) Type I Move:
c
ij
and
a
ij
both take us out of the current aspect
to a single aspect in each of the two directions—
a
ip
and
a
iq
,
respectively. We construct search tree nodes corresponding to
both moves.
2) Type II Move: Exactly one out of
c
ij
and
a
ij
takes us to a single
aspect
a
ip
. For the other direction, the aspect we would reach
depends upon the initial position
(
2
s
ij
;
e
ij
])
in the current as-
pect. We construct a search tree nodecorresponding to the former
move.
3) Type III Move: Whether we move in the clockwise or the coun-
terclockwise direction, the aspect reached depends on the initial
position in the current aspect. We choose the move which leads