IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 7, NO. 3, SEPTEMBER 2003 141
Computer-Aided Tumor Detection in Endoscopic
Video Using Color Wavelet Features
Stavros A. Karkanis, Member, IEEE, Dimitris K. Iakovidis, Dimitris E. Maroulis, Member, IEEE,
Dimitris A. Karras, Associate Member, IEEE, and M. Tzivras
Abstract—We present an approach to the detection of tumors
in colonoscopic video. It is based on a new color feature extraction
scheme to represent the different regions in the frame sequence.
This scheme is built on the wavelet decomposition. The features
named as color wavelet covariance (CWC) are based on the covari-
ances of second-order textural measures and an optimum subset
of them is proposed after the application of a selection algorithm.
The proposed approach issupported by a linear discriminant anal-
ysis (LDA) procedure for the characterization of the image regions
along the video frames. The whole methodology has been applied
on real data sets of color colonoscopic videos. The performance in
the detection of abnormal colonic regions corresponding to adeno-
matous polyps has been estimated high, reaching 97% specificity
and 90% sensitivity.
Index Terms—Color texture, computer aided colonoscopy,
image analysis, medical imaging, polyp detection, wavelet fea-
tures.
I. INTRODUCTION
C
OLORECTAL cancer is the second leading cause of
cancer-related deaths in the United States [1], [2]. More
than 130 000 people are diagnosed with colon cancer each year
and about 55 000 people die from the disease annually. Colon
cancer can be prevented and cured through early detection,
so early diagnosis is of critical importance role for patient’s
survival. Screening is the current and most suitable prevention
method for an early detection and removal of colorectal polyps.
If such polyps remain in the colon, they can possibly grow
into malignant lesions. Colonoscopy is an accurate screening
technique for detecting polyps of all sizes, which also allows
for biopsy of lesions and resection of most polyps [3]. The
colonic mucosal surface is granular and demarcated into small
areas called nonspecific grooves. Changes in the cellular pat-
Manuscript received March 6, 2002. This work was supported in part by the
Special Account of Research Grants, National and Kapodestrian University of
Athens.
S. A. Karkanis was with Realtime Systems and Image Analysis Group,
Department of Informatics and Telecommunications, University of Athens.
He is now with the Department of Informatics and Computer Technology,
Technological Educational Institute of Lamia, Lamia 35100, Greece (e-mail:
sk@teilam.gr).
D. K. Iakovidis and D. E. Maroulis are with Realtime Systems and
Image Analysis Group, Department of Informatics and Telecommunications,
University of Athens, 15784 Athens, Greece (e-mail: rtsimage@di.uoa.gr;
rtsimage@di.uoa.gr).
D. A. Karras is with Hellenic Aerospace Industry, Schematari, Greece
(e-mail: dkarras@haicorp.com).
M. Tzivras is with Gastroenterology Section, Department of Pathophysi-
ology, Medical School, University of Athens, Athens 11527, Greece (e-mail:
dkarras@haicorp.com).
Digital Object Identifier 10.1109/TITB.2003.813794
tern (pit pattern) of the colon lining might be the very earliest
sign of polyps. Pit patterns can be used for a qualitative and
quantitative diagnosis of lesions. These textural alterations of
the colonic mucosal surface can also be used for the automatic
detection of colorectal lesions [4]–[6].
The scope of this work is the location of regions suspicious
for malignancy in video colonoscopy, regions that require more
thorough examination by medical experts for a second evalua-
tion. Tumor detection schemes using textural information have
been proposed for various tissues such as liver [7], prostate [8],
breast [9], brain [10], cervix [11], and cardiac [12]. Automated
classification and identification of colonic carcinoma using mi-
croscopic images and involving texture analysis compared with
geometric features based on statistical analysis has been pro-
posed by Esgiar et al. [13], [14]. The use of endoscopic video
frames for the identification of adenomatous polyps involving a
novel wavelet based color texture analysis scheme is a topic that
it has not been reported in the literature to the best of our knowl-
edge. In the proposed approach the video frame sequences are
transformed in scale and frequency by using the wavelet trans-
form since it has been observed that the textural information is
localized in the middle frequencies and lower scales of the orig-
inal signal [15]. Statistical color wavelet features have been en-
countered in this texture analysis scheme, for the discrimination
of normal and abnormal (i.e., tumor) regions. The construction
of the texture feature space follows the multiresolution approach
on the color domain. The resulted space is found to be discrim-
inant. A linear classification scheme was used to label image
regions with a low error rate. The novel proposed color wavelet
textural features are favorably compared to the rival approach
of wavelet correlation signatures [16].
The proposed detection scheme involves a) a novel feature
extraction technique based on a discrete wavelet decomposition
applied on different color spaces and b) statistical analysis of
the wavelet coefficients associated with the color bands. The
wavelet features are based on second-order textural information
estimated on the domain of the discrete wavelet decomposition
of each color band of a video frame. In this paper, the textural
characteristics estimated on the color discrete wavelet frame
transform and in the sequel processed by using correlational
analysis, give valuable information about the set of features that
produce the most discriminant subspaces for normal/abnormal
tissue regions. The proposed scheme was tested on real data sets
of color colonoscopic videos provided by the Gastroenterology
Section, Department of Pathophysiology, Medical School, Uni-
versity of Athens, Greece, and partially by the Section for Min-
imal Invasive Surgery, University of Tübingen, Germany. The
1089-7771/03$17.00 © 2003 IEEE
142 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 7, NO. 3, SEPTEMBER 2003
video sequences used for evaluation were selected to contain
relatively small polyps, as physicians suggested. The sequences
were evaluated by endoscopy experts and compared with the
corresponding histological results, proving the accuracy of the
proposed methodology (this evaluation procedure with respect
to the histological data led to specificity ranging from 86% to
98% and sensitivity ranging from 79% to 96.5%).
The rest of the paper is organized as follows. Medical infor-
mation on colorectal polyps is provided in Section II. In Sec-
tion III, the fundamental properties of color and texture analysis
involved along with the proposed methodology are presented.
Section IV describes the evaluation approach and the results ob-
tained from the extensive experimentation. Finally, discussion
of the results as well as the conclusions of this study is presented
in Sections V and VI, respectively.
II. M
EDICAL BACKGROUND
A polyp is defined as any visible tissue mass protruding
from the mucosal surface. Polyps are characterized according
to their color, appearance of their mucosal surface, presence
of ulcers, their bleeding tendency, and above all the presence
of pedunculus (pedunculated or nonpedunculated). Their size
varies from barely visible transparent protrusions to penducu-
lated lesions with a diameter of 3 to 5 cm. Although there are
many histopathologic types of polyps, the majority of them
are adenomatous. Approximately 75% of the colonic polyps
are adenomatous [17]. Adenomatous polyps are neoplasms
that result from disordered cell proliferation, differentiation,
and apoptosis [18]. The evolution of an adenomatous polyp
to cancer is the result of a multistep process that involves
many molecular and genetic mechanisms including activation
of oncogenes and suppression of tumor genes [19]. The real
prevalence of colonic polyps in the general population is not
known. Polyps may be found in the colon of 30%–50% of
people older than 55 years old, while colonoscopy surveys
showed a lower incidence, at the level of 30% [20]. Today, the
international consensus for the treatment of polyposis dictates
removal of all polyps, regardless of the location, size or other
characteristics, in order to prevent a possible development
to cancer. Colonoscopy remains the best available procedure
to detect polyps, with many advantages such as the ability
to have simultaneous tissue biopsy or polypectomy [3]. A
competitive new generation technique used for the detection
of colorectal polyps is virtual colonoscopy based on computer
tomography (CT) or magnetic resonance (MR) data. This
technique utilizes specialized imaging software that allows for
a three-dimensional visualization of the colon and the rectum
by combining multiple volumetric tomographic data [21]–[24].
It has the advantage that it does not discomfort the patients
as the standard colonoscopy, but it is not so accurate for the
detection of small lesions and it can not easily discriminate
polyps among retained stool or thickened folds because they
can mimic their shape and density and does not allow for tissue
biopsy or polypectomy [23], [24].
Important research on the automated detection of polyps on
virtual colonoscopy data has been reported in the recent litera-
ture. Most of this research was concentrated on the use of geo-
metric features for the discrimination of polyps from normal
colonic regions [24]–[27].
III. C
OLOR TEXTURE ANALYSIS
Color texture analysis is based on the combined information
from both color and texture fields of the image. Texture
processing was mainly focused on the use of gray-level image
information for a number of years [28], [29]. Pioneering
studies exploiting the combination of both color and texture
information, have been presented by Caelli and Raye [31],
Sharkanski et al. [32], and Kondepudy et al. [33]. More recent
studies involving color texture analysis, include the calculation
of chromaticity moments [34], a perceptual approach for the
segmentation of color textures [35], Gabor filtering of complex
hue/saturation images [36], moving average modeling [37] and
color and texture fusion by combining color and multireso-
lution simultaneous autoregressive models [38]. Drimbarean
and Whelan [39] performed experiments using grayscale and
color features based on discrete cosine transform, Gabor and
cooccurrence matrices in different color spaces. The results
of this study led to the conclusion that the introduction of
color information, especially by calculating grayscale texture
features on the different color channels, significantly improves
color texture classification. Other approaches that have taken
into account the correlation of texture measures between
the different color channels, have shown that color texture
information can also be found in the way color channels are
related to each other. Under this framework Paschos [40]
proposed a set of discriminative and robust chromatic correla-
tion features using directional histograms, Van de Wouwer et
al. [41] achieved high classification results using correlation
signatures calculated on the wavelet coefficients of the different
color channels of the images and Vandenbroucke et al. [42]
exploited the correlation of first-order statistical features
among the different color channels for unsupervised soccer
image segmentation. In this work, we propose the covariance
of second-order statistical features in the wavelet domain for
the characterization of colonic polyps.
A. Color Spaces
Color is a property of the brain and not of the outside world
[43]. The nervous system, instead of analyzing colors, uses the
information of the external environment, namely the reflectance
of different wavelengths of light and transforms this information
into colors [44]. The use of the red-green-blue (RGB) space is
very common in image and video-processing research, dictated
primarily by the availability of such data as they are produced
by most color image-capturing devices. Drawbacks in the use of
RGB in computer vision applications are: the high correlation
among RGB channels for natural images [45], therepresentation
of RGB is not very close to the way humans perceivecolors [46]
and it is not perceptually uniform [47].
In RGB space, each color is represented as a triple (R, G, B),
where R, G, and B represent red, green, and blue signals corre-
sponding to different wavelengths of the visible spectrum. As-
suming dichromatic reflection and white illumination, a color
transform that is independent of the viewpoint, surface orienta-
tion, illumination direction, and illumination intensity, has been
KARKANIS et al.: COMPUTER AIDED TUMOR DETECTION IN ENDOSCOPIC VIDEO 143
proposed by Gevers [48]. A first-order invariant instantiation of
this transform, which is also more robust to noise comparing to
other invariant instantiations, has proven to be the normalized
RGB space (Appendix I-A). Normalized
has been used for
automatic lip reading [49] and other face detection applications
[52].
A variety of other color spaces are used in different appli-
cations. The international committee on colorimetry, Commis-
sion Internationale de l’Eclairage (CIE), established the XYZ
color space as standard, based on the response curves of the
eyes and statistics that were performed on human observers
[47]. Normalizing the XYZ (Appendix I-B), the occurring
space, has been proven to be noise robust for texture recognition
using chromatic correlation features [40]. All of the above color
spaces have the advantage of isolating the luminance compo-
nent
from the two-chrominance components [53].
The Karhunen–Loeve (K–L) transformation applied on im-
ages, has been proved to be best for color texture characteriza-
tion as reported by Van De Wouver et al. [41], and for the anal-
ysis of skin lesions [54]. K-L transform is formed by the eigen-
vector of the correlation matrix of an image, which remains ap-
proximately the same for a largeset of natural color images [41],
[55]. It transforms an image to an orthogonal basis in which the
axes are statistically uncorrelated. In that sense, the information
presented in RGB space is decorrelated. Practically, it that can
be produced as a linear transformation of the RGB coordinates
(Appendix I-C).
Perceptual uniformity has been considered to form color
spaces that describe color similarity to the way humans perceive
color. Generally, a system is perceptually uniform if a small
perturbation to a component value is approximately equally
perceptible across the range of that value [47]. CIE-Lab is a
perceptually uniform color space that has proved to perform
better than
for color texture analysis, but not in the
presence of noise [53]. It has been applied for several color
texture classification tasks such as: the retrieval of color
patterns using textural features [56], analysis of skin lesions
[54] and segmentation of human flesh [57], with a performance
that has been considered high. The coordinates of CIE-Lab as
a function of
are given in Appendix I-D.
Another, approximately perceptually uniform color space is
defined in terms of hue, saturation and value (HSV), a phenom-
enal color space[58]. Phenomenal color spaces attempt to clas-
sify colors in relation to how they are perceived and interpreted
by the human brain and they are more “intuitive” in manipu-
lating color. HSV has led to higher classification performance
than CIE-Lab and RGB in both noisy and noise-free conditions
for color texture analysis [53]. On the other hand, Palm et al.
[36] showed that HSV performs equivalently to
for color
texture classification using different features. Another common
alternative similar to HSV is hue, lightness, saturation (HLS)
space [46], [59]. HLS has been applied to represent the color of
the tongue for medical diagnosis [60].
B. Second-Order Statistics on the Wavelet Domain as
Grayscale Textural Features
As it has already been noted, the size of the lesions to be de-
tected using the proposed framework varies. The image resolu-
tion cannot be defined so as to cover the majority of the lesions
sizes. It will be useful to face the problem in a way that detects
the information in different resolutions by exploiting the inter-
mediate scales for the final decision. Multiresolution analysis of
an image can be achieved by using the discrete wavelet trans-
form.
Texture is the discriminating information that differentiates
normal from abnormal lesions [4]–[6]. Since texture is essen-
tially a multiscale phenomenon, multiresolution approaches
such as wavelets perform well for texture analysis. A character-
ization of texture is usually based on the local information that
appears within a neighborhood distribution of the gray levels.
The proposed methodology focuses on a single scale in order
to extract the relevant information. Recent studies have come
to the conclusion that a spatial/frequency representation, which
preserves both global and local information, is adequate for
the characterization of texture. The wavelet transform offers a
tool for spatial/frequency representation by decomposing the
original images to the corresponding scales. When decompo-
sition level decreases in the spatial domain, it increases in the
frequency domain providing zooming capabilities and local
characterization of the image. Since the low-frequency image
produced by the transformation does not contain major texture
information and the most significant information of a texture
often appears in the middle-frequency channels, we choose to
use discrete wavelet transform (DWT) for the decomposition
of the frequency domain of the image [61], [16], [63]. Wavelet
frame representation of the image offers a representation of the
frequency domain. Such representations have been proposed
because they have greater robustness in the presence of noise,
can be sparser, and can have greater flexibility in representing
the structure of the input data. The dimensionality and the
representation of input is not a unique combination of basis
vectors. The two-dimensional (2-D) DWT transformation is
implemented by applying a separable filterbank to the image
[64].
This filtering procedure convolves the image with a lowpass
and bandpass filter , which produces a low-resolution
image
at scale and the detail images at scale
. The repetition of this filtering procedure
results in a decomposition of the image at several scales. The
final set consisting of the low resolution image
and all the
detailed images
along the scale is the multiscale
representation of the image at a specific depth defined by
the total number of scales. This filtering procedure can be
described by the following recursive equations [16]:
(1)
where the arrow
denotes the subsampling procedure, the
asterisk is the convolution operator, and and are the
two filters for all
.
The cooccurrence matrices approach has been considered
in this work for the description of a statistical model of the
texture encoded within the decomposed subimages. It captures
second-order gray-level information, which is mostly related
to the human perception and discrimination of textures [65].
For a coarse texture these matrices tend to have higher values
144 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 7, NO. 3, SEPTEMBER 2003
near the main diagonal whereas for a fine texture the values
are scattered. The cooccurrence matrices encode the gray level
spatial dependence based on the estimation of the second-order
joint-conditional probability density function
,
which is computed by counting all pairs of pixels at distance
having gray levels and at a given direction . The angular
displacement is usually included in the range of the values
. Among the 14 statistical measures, orig-
inally proposed by Haralick [28], [66], that derive from each
cooccurrence matrix we consider only four. Namely, angular
second moment, correlation, inverse difference moment and
entropy
(2)
(3)
(4)
(5)
where
is the th entry of the normalized cooccurrence
matrix,
is the number of gray levels of the image, and
and are the means and standard deviations of the
marginal probability
obtained by summing up the rows
of matrix
. These measures provide high discrimination
accuracy which can be only marginally increased by adding
more measures in the feature vector [67].
In addition to the features (2)–(6), Esgiar et al. proposed the
use of contrast [14] or the use of both contrast (also known
as difference moment) and dissimilarity [13], for microscopic
image analysis of colonic tissue
(6)
(7)
In Section IV, we experimentally show that the use of these fea-
tures do not provide additional textural information that is sig-
nificant for the analysis of the macroscopic video images used
in our application.
C. Second-Order Color Wavelet Covariance (CWC) Features
The proposed approach is based on the extraction of color
textural features. These features are estimated over the second-
order statistical representation of the wavelet transform of the
color image. Since each feature represents a different property
of the examined region, we consider as valuable information
the covariance among the different statistical valuesbetween the
color channels of the examined region.
According to the definition of texture, it is mainly related to
the distribution of the intensities [28], [61]. It is then expected
that similar textures will have close statistical distributions and
consequently they should appear to have similar feature values
of the features. This similarity property of the selected features
can be described by measuring the variance in pairs of them.
By using the covariance between two features, we can have a
measure of their “tendency” to vary together. The texture co-
variance has been proposed in the literature [29] as a measure
that is used directly on the image intensities or among the color
intensities of the examined region. Our method uses the covari-
ance in order to rank the changes in the statistical distribution
of the intensities between the examined regions in the different
color channels. By noticing the way the features of examined
texture regions covary it will be an easy task to decide if they
belong to the same texture class since in similar textures we ex-
pect measures to covary.
By considering the original image
, we obtain its color trans-
formation. Color transformations result in three decomposed
color channels
(8)
A three-level discrete wavelet frame transformation is conse-
quently applied on each color channel
. This transformation
results in a new representation of the original image, according
to the corresponding equations of wavelet decomposition (1).
This decomposition procedure produces a low-resolution image
at scale and the detail images and .
In our case, we have
(9)
where
is the decomposition level.
Since the textural information is better presented in the
middle wavelet detailed channels, we consider the second level
detailed coefficients. Thus, the image representation that is
finally considered is the one consisting of the detail images
produced from (9) for the values
. This results in a
set of nine different subimages
(10)
For the extraction of the second-order statistical textural infor-
mation, we use cooccurrence matrices calculated over the above
nine different subimages. These matrices reflect the spatial in-
terrelations between the intensities within the wavelet decompo-
sition level. The cooccurrence matrices are estimated in four dif-
ferent directions of intensities’ relation, 0
,45 ,90 , and 135 ,
resulting to 36 matrices
(11)
Finally the four statistical measures, namely angular second
moment, correlation, inverse difference moment, and entropy
are estimated for each matrix resulting in 144 wavelet features
(12)
where
is the respective statistical measure.
In the proposed scheme, we consider as a textural measure
the covariance of the same statistical measure be-
KARKANIS et al.: COMPUTER AIDED TUMOR DETECTION IN ENDOSCOPIC VIDEO 145
tween color channels at wavelet band which is defined ac-
cording to the following equations:
(13)
where
represents the different angles for the cooccurrence
matrices
.
Since the covariance (13) relates pairs of features, the
proposed set of features is a set of 72 components. The 36 of
them are the variances as they relate features of the same color
channel and the rest 36 represent features of different color
channels estimated by the corresponding covariance values.
We call this set of the 72 components color wavelet covariance
features, the CWC feature vector.
The extraction of the CWC vector can be described in the
following steps.
a) The original color image (video frame) is decomposed
into three separate color bands.
b) Each band is scanned across with fixed size sliding square
window.
c) Each window is then transformed according to a three-
level 2-D discrete wavelet transform by using decompo-
sition functions that follow the properties of the wavelet
frames. The detail coefficients of the middle decomposi-
tion level are considered for further processing. This step
results to a set of nine subimages.
d) The cooccurrence matrices, for each image of the previous
step, are estimated into four directions, producing 36 ma-
trices that are a second-order statistical representation of
the original image.
e) Four statistical measures (angular second moment, en-
tropy, inverse differencemoment, and correlation) are cal-
culated for each matrix, resulting in a set of 144 compo-
nents. Each of the measure carries different information
about the texture.
f) Covariance values of pairs of the estimated features
(e) constitute the 72-dimensional CWC feature vector,
to be used for the classification of the image regions
(windows).
IV. E
XPERIMENTS AND RESULTS
The experimental study of this paper outlines the series of the
conducted experiments and the obtained results in order to eval-
uate the proposed novel feature-extraction methodology, along
with its associated parameters in the problem of tumor detection
using color colonoscopic video sequences.
A. Data Acquisition and Processing
The colonoscopic data used in the following experiments was
acquired from different patients with an Olympus CF-100 HL
Fig. 1. Three-level wavelet decomposition scheme of the original image for
color channel
i
.
TABLE I
H
ISTOLOGICAL CHARACTERIZATION OF THE AVAILABLE DATASET
endoscope. The major interest for the tumor detection problem,
as the experts have suggested it, has led us to the use of video
frames mainly of small size adenomatous polyps. Since they are
not easily detectable, they are more common and more likely to
become malignant compared to the hyperplastic polyps [17].
Sixty-six patients having relatively small polyps were exam-
ined within a period of eight months. The results of the histo-
logical evaluation of these polyps are presented in Table I [62].
The mean diameter of the adenomatous polyps was estimated to
be
mm. A total number of 60 video sequences cor-
responding to the different adenomas with a duration ranging
between 5 to 10 s, were used for the evaluation of the proposed
methodology. The video frame sequences were recorded during
the clinical examination of the patients and then digitized by
using a commercial RGB-color frame grabber at a rate of 25
frames per second, a resolution of 1 K
1 K pixels and 24 bits
per pixel color depth (eight bits for each color channel). Each
one of these video frame sequences, selected by the physician
as indicative cases, illustrates small size lesions of interest at
different position, scale and lighting conditions (Fig. 2). In the
experiments outlined in the following section training and test
set of frames have been considered. The training set comprised
of 180 frame images (up to three frames per video sequence)
shown by the experts group. The selection of the frame images
to be incorporated in the training set has been very carefully per-
formed by the experts group in order to minimize the bias intro-
duced in the training procedure. Each one of the five members of
the expert group has independently selected 200 image frames
as representative of the image frames encountered in the normal
subject of the video-sequences as well as 400 image frames as
