IEEE TRANSACTIONS ON IMAGE PROCESSING 1
Directional Multiscale Modeling of Images
using the Contourlet Transform
Duncan D.-Y. Po and Minh N. Do
Coordinated Science Lab and Beckman Institute
University of Illinois at Urbana-Champaign
Urbana IL 61801
Email: duncanpo@ifp.uiuc.edu, minhdo@uiuc.edu
Abstract
The contourlet transform is a new extension to the wavelet transform in two dimensions using
nonseparable and directional filter banks. The contourlet expansion is composed of basis images oriented
at varying directions in multiple scales, with flexible aspect ratios. With this rich set of basis images, the
contourlet transform can effectively capture the smooth contours that are the dominant features in natural
images with only a small number of coefficients. We begin with a detailed study on the statistics of the
contourlet coefficients of natural images, using histogram estimates of the marginal and joint distributions,
and mutual information measurements to characterize the dependencies between coefficients. The study
reveals the non-Gaussian marginal statistics and strong intra-subband, cross-scale, and cross-orientation
dependencies of contourlet coefficients. It is also found that conditioned on the magnitudes of their
generalized neighborhood coefficients, contourlet coefficients can approximately be modeled as Gaussian
variables. Based on these statistics, we model contourlet coefficients using a hidden Markov tree (HMT)
model that can capture all of their inter-scale, inter-orientation, and intra-subband dependencies. We
experiment this model in the image denoising and texture retrieval applications where the results are
very promising. In denoising, contourlet HMT outperforms wavelet HMT and other classical methods in
terms of visual quality. In particular, it preserves edges and oriented features better than other existing
methods. In texture retrieval, it shows improvements in performance over wavelet methods for various
oriented textures.
January 22, 2004 DRAFT
2 IEEE TRANSACTIONS ON IMAGE PROCESSING
I. INTRODUCTION
In image processing, it has been a common practice to use simple statistical models to describe images.
Natural images tend to have certain common characteristics that make them look “natural.” The aim of
statistical modeling is to capture these defining characteristics in a small number of parameters so that
they can be used as prior information in image processing tasks such as compression and denoising. A
simple, accurate and tractable model is an essential element in any successful image processing algorithm.
Images can be better modeled in the wavelet transform [1],[2] domain, which shows multiresolution
and time-frequency localization properties, and in which energy density has a more local structure than
in the image spatial domain. Initially, wavelet transform was considered to be a good decorrelator for
images, and thus wavelet coefficients were assumed to be independent and were simply modeled by
marginal statistics [3]. Later it was realized that wavelet coefficients for natural images exhibit strong
dependencies both across scales and between neighbor coefficients within a subband, especially around
image edges. This gave rise to several successful joint statistical models in the wavelet domain [4]–[10],
as well as improved image compression schemes [11]–[13].
The major drawback for wavelets in 2-D is their limited ability in capturing directional information. To
counter this deficiency, researchers have most recently shifted their attention to multiscale and directional
representations that can capture the intrinsic geometrical structures such as smooth directional contours
in natural images. Some examples include the steerable pyramid [14], brushlets [15], complex wavelets
[16], and the curvelet transform [17]. In particular, the curvelet transform, pioneered by Cand`es and
Donoho, was shown to be optimal in a certain sense for functions in the continuous domain with curved
singularities.
Inspired by curvelets, Do and Vetterli [18]–[20] developed the contourlet representation based on an
efficient two-dimensional nonseparable filter bank that can deal effectively with images having smooth
contours. Contourlets not only possess the main features of wavelets (namely, multiresolution and time-
frequency localization), but also show a high degree of directionality and anisotropy. The main difference
between contourlets and other multiscale directional systems is that contourlets allow for a different and
flexible number of directions at each scale, while achieving nearly critical sampling. In addition, contourlet
transform employs iterated filter banks, which makes it computationally efficient.
In this work, we focus on image modeling in the contourlet domain. Our primary goal is to provide
an extensive study on the statistics of contourlet coefficients in order to gain a thorough understanding of
their properties. Then we develop an appropriate model that can capture these properties, which can be
DRAFT January 22, 2004
PO AND DO: DIRECTIONAL MULTISCALE MODELING OF IMAGES USING THE CONTOURLET TRANSFORM 3
useful in future contourlet applications, including compression, denoising, and feature extraction. Similar
to wavelet-based models, contourlet-based models need to take into account the coefficients’ dependencies
across scale and space. However, as a “true” two-dimensional representation, contourlets allow us to also
model the coefficients’ dependencies across directions. In other words, contourlet modeling allows us to
jointly model all three fundamental parameters of visual information, namely: scale, space, and direction.
The rest of the paper is organized as follows. Section II introduces the basics of contourlets including
their transform algorithm, structure, properties, and coefficient relationships. In Section III, we study
the marginal and joint statistics of contourlet coefficients of natural images via histograms. Section IV
examines the dependencies between coefficients using mutual information. Inspired by these results, we
develop a hidden Markov tree (HMT) model for the contourlet transform in Section V. In Section VI,
we apply the contourlet HMT model in denoising and texture retrieval. Finally, a conclusion is presented
in Section VII.
II. BACKGROUND
A. Contourlets
Do and Vetterli developed contourlets in [18]–[20]. Their primary aim was to construct a sparse efficient
decomposition for two-dimensional signals that are piecewise smooth away from smooth contours. Such
signals resemble natural images of ordinary objects and scenes, with the discontinuities as boundaries of
objects. These discontinuities, referred to as edges, are gathered along one-dimensional smooth contours.
Two-dimensional wavelets, with basis functions shown in Figure 1(a), lack directionality and are only
good at catching zero-dimensional or point discontinuities, resulting in largely inefficient decompositions.
For example, as shown in Figure 1(c), it would take many wavelet coefficients to accurately represent
even one simple one-dimensional curve.
Contourlets were developed as an improvement over wavelets in terms of this inefficiency. The resulting
transform has the multiresolution and time-frequency localization properties of wavelets, but also shows a
very high degree of directionality and anisotropy. Precisely, contourlet transform involves basis functions
that are oriented at any power of two’s number of directions with flexible aspect ratios, with some
examples shown in Figure 1(b). With such richness in the choice of basis functions, contourlets can
represent any one-dimensional smooth edges with close to optimal efficiency. For instance, Figure
1(d) shows that compared with wavelets, contourlets can represent a smooth contour with much fewer
coefficients.
January 22, 2004 DRAFT
4 IEEE TRANSACTIONS ON IMAGE PROCESSING
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250
50
100
150
200
250
ContourletWavelet
(a) (b) (c) (d)
Fig. 1. Contourlet and wavelet representation for images. (a) Basis functions of 2-D wavelets (b) Basis functions of contourlets
(c) Wavelets have square supports and can only capture points. (d) Contourlets have elongated supports and can capture line
segments. Contourlets thus can effectively represent a smooth contour with fewer coefficients.
(2,2)
multiscale dec. directional dec.
(-pi,-pi)
(pi,pi)
w1
w2
(a) (b)
Fig. 2. (a) Pyramidal directional filter bank structure that implements the discrete contourlet transform. (b) A typical contourlet
frequency partition scheme.
Contourlets are implemented by the pyramidal directional filter bank (PDFB) which decomposes images
into directional subbands at multiple scales [18]–[20]. The PDFB is a cascade of a Laplacian pyramid
[21] and a directional filter bank [22] as shown in Figure 2(a). The directional filter bank is a critically
sampled filter bank that decomposes images into any power of two’s number of directions. Due to
the PDFB’s cascaded structure, the multiscale and directional decompositions are independent of each
other. One can decompose each scale into any arbitrary power of two’s number of orientations and
different scales can be divided into different numbers of orientations. This decomposition property makes
contourlets a unique transform that can achieve a high level of flexibility in decomposition while being
DRAFT January 22, 2004
PO AND DO: DIRECTIONAL MULTISCALE MODELING OF IMAGES USING THE CONTOURLET TRANSFORM 5
(v)
(iv)
(iii)
(ii)
(i)
(a) (b)
(v)
(iv)
(iii)
(ii)
(i)
(c) (d)
Fig. 3. (a) The “Peppers” image. (b) Contourlet representation of (a). (c) The “Goldhill” image. (d) Contourlet representation of
(c). (i)–(v) represents coarse to fine scales respectively. Small coefficients are colored black while large coefficients are colored
white.
close to critically sampled (up to 33% overcomplete, which comes from the Laplacian pyramid)
1
. Other
multiscale directional transforms have either a fixed number of directions, such as complex wavelets [16],
or are significantly overcomplete (depending on the number of directions), such as the steerable pyramid
[14]. Figure 2(b) shows a typical frequency division of the contourlet transform where the four scales
are divided into four, four, eight and eight subbands from coarse to fine scales respectively. The fact that
contourlets are close to critically sampled makes them especially promising in image compression.
Figure 3(a) shows the image “Peppers” and Figure 3(b) shows its contourlet representation. Similarly,
Figure 3(c) shows the image “Goldhill” and Figure 3(d) shows its contourlet representation. In this
particular decomposition, the image is divided into an approximation image (i) and four detail scales
(ii), (iii), (iv), (v) from coarse to fine. Each detail scale is further partitioned into directional subbands
according to the scheme in Figure 2(b). The two coarser scales are partitioned into four directional
1
Recently, a modified version of the contourlet scheme that is critically sampled was developed [23].
January 22, 2004 DRAFT
