COPYRIGHT
NOTICE
FedUni ResearchOnline
http://researchonline.ballarat.edu.au
This is the submitted for peer-review version of the following article:
Kulkarni, S., & Kulkarni, P. (2012). Colour image annotation using hybrid
intelligent techniques for image retrieval. Hybrid Intelligent Systems. 115-119
Which has been published in final form at:
http://dx.doi.org/10.1109/HIS.2012.6421319
© 2012 IEEE.
This is the author’s version of the work. It is posted here with permission
of the publisher for your personal use. No further distribution is permitted.
MapReduce Neural Network Framework for Efficient Content Based Image
Retrieval from Large Datasets in the Cloud
Sitalakshmi Venkatraman
School of Science, Information Technology and
Engineering, University of Ballarat
Ballarat, Australia
e-mail: s.venkatraman@ballarat.edu.au
Siddhivinayak Kulkarni
School of Science, Information Technology and
Engineering, University of Ballarat
Ballarat, Australia
e-mail: s.kulkarni@ballarat.edu.au
Abstract—Recently, content based image retrieval (CBIR) has
gained active research focus due to wide applications such as
crime prevention, medicine, historical research and digital
libraries. With digital explosion, image collections in
databases in distributed locations over the Internet pose a
challenge to retrieve images that are relevant to user queries
efficiently and accurately. It becomes increasingly important
to develop new CBIR techniques that are effective and scalable
for real-time processing of very large image collections. To
address this, the paper proposes a novel MapReduce neural
network framework for CBIR from large data collection in a
cloud environment. We adopt natural language queries that
use a fuzzy approach to classify the colour images based on
their content and apply Map and Reduce functions that can
operate in cloud clusters for arriving at accurate results in
real-time. Preliminary experimental results for classifying and
retrieving images from large data sets were quite convincing to
carry out further experimental evaluations.
Keywords-CBIR; image retrieval, neural network,
MapReduce, cloud
I. INTRODUCTION
Content-based image retrieval (CBIR) is a technique that
involves retrieving specific images from image databases
primarily based on features that could be automatically
derived [1]. Typically, primitive features such as colour,
texture, object or shape are used, and could be combined
with logical features such as the identity of the objects in the
image, or even abstract attributes such as the significance or
relevance of the image within a context [2]. With the rising
popularity of social media and the prevalence of mobile
image capturing devices, the amount of images available in
the digital media and databases has grown at an exponential
rate [3][4]. Current CBIR systems are not capable of
catering to high level user demands that are based on natural
language queries, and the existing low level retrieval
techniques adopted, based on colour, texture, shape and
object, are highly inefficient with the exponential growth of
large datasets.
Image databases could contain
tainment, and sports
fields [5][6]. There is always a need to find a specific image
from a large collection that could be shared by many
professional groups or even available freely in the Internet.
Searching for relevant content in large collections of image
databases has become a difficult process [2][3].
Traditionally, such CBIR is mostly limited through a search
using tags or keywords assigned with the image while
storing in the databases. However, if the image is not
uniquely tagged or described wrongly, the search results
obtained will be of little value to the users. Hence, for
accurate results, most of the CBIR systems use query
images as examples for matching and retrieving the desired
image from a digital collection [3][4][7]. In most of the
situations, query images may not be available for the search,
and users are looking for flexible and intuitive ways of
image retrieval [8][9]. In addition, current CBIR techniques
are computationally intensive for achieving high accuracies,
and hence they are inefficient for real-time image retrieval
from very large databases.
This paper proposes a novel CBIR technique of using
natural language queries to retrieve images more accurately
and efficiently, by adopting a MapReduce neural network
framework for processing from very large image databases
available in today's cloud environment. This technique
overcomes the limitations of current CBIR systems that can
operate only at the primitive feature level as users are given
freedom to pose their queries in terms of natural language
[10][11][12]. We combine image colours such as red, blue
and green, and content types such as low, medium, high and
very high in natural language queries. Our proposed
technique performs fusion of such queries using neural
networks. The computational intensive processing that could
drastically slow down the search performance for a large
media collection is overcome by parallelising using
MapReduce framework [13]. The parallelism is achieved by
splitting the process into many smaller sets of independent
tasks and the concept of neural network ensemble (NNE)
has been used to implement the fusion of classes that are
formed based on colour and content specified by users in the
form of natural language queries. Since Neural Network
Ensembles (NNE) divide the data into smaller areas for
faster learning of each class, it is more efficient than single
neural network for processing large amount of data and
therefore is a suitable candidate for applying the
MapReduce framework such as Hadoop [14] to speed up the
63
978-1-4673-5115-7
c
2012 IEEE
calculations and return the search results at a shorter time.
The performance improvements gained from computing the
process in a parallel manner within the MapReduce
framework results in real-time efficiency even when scaled
to very large image collections occupying petabytes of
storage in the cloud.
The rest of paper is organized as follows. Section 2
provides a literature review of CBIR systems. Section 3
describes the proposed CBIR technique in a novel
MapReduce neural network framework for large image
databases. Section 4 presents the experimental results of our
CBIR technique. Finally, conclusions and future work of
this research are given in section 5.
II. R
EVIEW OF CBIR SYSTEMS
CBIR systems developed earlier such as QBIC [15],
Virage [16], Photobook [17] and Netra [18] have used a
weighted linear method to combine similarity measurements
of different feature classes. However, more sophisticated
techniques have recently been introduced.
The main motivation for the development of this system
is that region-based search improves the quality of the
image retrieval. Therefore the system incorporates an
automated region identification algorithm. There are other
existing techniques in the literature for content-based
retrieval such as satellite image browsing using automatic
semantic categorization [19], partial relevance in interactive
facial image retrieval [20] and region-based image
clustering and retrieval using multiple instance learning [21].
A detailed survey published in 2005 on content-based image
retrieval can be found in [22].
Computational intelligence based techniques such as
neural networks have also been applied by some researchers
to develop a prototype for CBIR. Neural networks have also
been proposed for feature extraction [23], similarity
measurement [24], relevance feedback technique [25].
Images are compared through a weighted dissimilarity
function which can be replaced as a “network of
dissimilarities.” The weights are updated via an error back-
propagation algorithm using the user’s annotations of the
successive set of the result images [26]. It allows an iterative
refinement of the search through a simple interactive
process. Not much work has been done in the area of fuzzy
logic based linguistic queries for image retrieval. Fuzzy
logic has impressive power to represent the queries in terms
of natural content of the image.
In [27] an image is represented by a set of segmented
regions each of which is characterised by a fuzzy feature
reflecting colour, texture and shape properties. The
resemblance between two images is then defined as the
overall similarity between two families of fuzzy features
and quantified by unified feature matching. Non-Boolean
fuzzy and similarity predicates are used to rank tuples
according to fuzzy based algebra [10]. Soft queries in image
retrieval systems present the use of soft computing and user
defined classifications in multimedia database systems for
content based queries [28]. A CBIR system which
automatically clusters images using features of those images
which are fuzzy in nature. Image is described on a fuzzy
rule based compact composite descriptor which includes
global image features combining brightness and texture
characteristics [29]. Paper presented by Yu and Dunham
[30] proposed fuzzy logic based method to automatically
generate the description of spatial relationship among
objects and graph based fuzzy linguistic metadata schema
for topology and relationship for a set of objects.
In [31], fuzzy logic is imported into image retrieval
phase to deal with the vagueness and ambiguity of human
judgment of image similarity by adopting the fuzzy
language variables to describe the similarity degree of
image features, not the features themselves. The fuzzy
inference is then used to instruct the weight assignments
among various image features. In [32] paper presents a
fuzzy-neural approach for interpretation and fusion of
colour and texture features for CBIR systems. Similarly, a
novel fuzzy approach is proposed to classify the colour
images based on their content, and to pose a query in terms
of natural language for fast and efficient retrieval [33]. In
this paper, we advance such an approach further by
proposing a MapReduce neural network framework for a
real-time efficient CBIR system that can process petabytes
of data from very large image databases. We make use of
cloud clusters for introducing parallel processing to achieve
the desired accuracy and time efficiency that is warranted in
real-life applications of CBIR.
III. P
ROPOSED MAPREDUCE NEURAL NETWORK
FRAMEWORK FOR CBIR
A. What is MapReduce?
MapReduce is a distributed computing framework to
support parallel computations over large datasets in multiple
petabytes of storage available on clusters of computers. The
framework also has the advantage of parallel processing of
such Big Data on large clusters of commodity hardware in a
reliable, fault-tolerant manner.
The concept originates from map and reduce functions
commonly used in functional programming like Lisp, and
have been improvised in MapReduce framework, which
transform a list of pairs <key, value> into a list of values.
The Map functions are invoked in a distributed environment
across multiple machines by automatically partitioning the
input data into a set of K splits. These partitioned input sets
can be processed in parallel on different machines. Reduce
functions are invoked in a distributed environment by
partitioning the intermediate key space into P pieces using a
partitioning function such as a hashing function, hash(key)
mod P.
The MapReduce framework consists of a single master
Job Tracker and one slave Task Tracker per cluster node.
The master is responsible for scheduling the jobs' component
tasks on the slaves, monitoring them and re-executing the
64 2012 12th International Conference on Hybrid Intelligent Systems (HIS)
failed tasks.. This way reliability and fault tolerance is taken
care of. Figure 1 shows an example of high level
architecture of the MapReduce Framework implemented in
Hadoop with a cluster setup consisting of 1 master node and
2 slave nodes. Image files and their extracted features are
stored in the DataNodes by utilizing Hadoop Data File
System (HDFS). This would facilitate batch processing of
the files and features during MapReduce search tasks that are
assigned by the job tracker to the task tracker present in each
of the nodes to facilitate parallel processing in a cluster
environment.
Figure 1. Architecture of a Cluster Setup for MapReduce
B. Neural Network Ensembles for CBIR
The shortcomings and problems encountered with
traditional methods of image retrieval have led to the rise of
interest in CBIR techniques. At present there are some
commercial products like QBIC [15], Virage [16],
Photobook [17] and Netra [18]. As mentioned earlier, they
combine similarity measurements of different feature
classes using a weighted linear method. Such techniques
lack high level processing as they are not sophisticated
enough to handle real life multiple natural language queries
from users [32][33][34], and more importantly in efficiently
searching from very large and diverse image datasets. To
address these limitations, we have adopted a hybrid
technique for CBIR based on fuzzy logic and neural
networks within a MapReduce distributed framework for
processing very large image collections in the cloud.
In this proposed framework, natural language queries
posed by the user are processed in parallel to retrieve
relevant images from large data collections. . We combine
the colour features such as Red, Orange, Yellow, Green,
Cyan, Blue, Purple, Magenta, Pink, Black, White and Grey,
as well as fuzzy terms of colour content such as 'no colour',
'very low', 'low', 'medium', 'high', and 'very high' for colour
image classification and perform retrieval using neural
networks. This involves steps which include fuzzy
interpretation of user queries, neural network to train the
queries and a technique for the fusion of multiple queries.
The most important step of training the classes for the
fusion of queries facilitates in accuracy of results and we
adopt a neural network based technique for this purpose. We
adopt supervised learning neural network to efficiently learn
the colours and content types and a neural network
ensembles (NNE) for the fusion of classes. In situations that
require processing of large image datasets, NNE is produces
more accurate outcomes as they are robust and efficient than
single neural network.
The advantage of NNE is that different networks such as
multilayer perceptrons, radial basis functions neural
networks, and probabilistic neural networks
, can take as
inputs samples characterised by different feature of colour
and content type. Feature set
for the colour Magenta in
an image is given by the following formula:
where,
represents the number of magenta colour pixels
in the image. and
represents the total number of pixels
in the image. Figure 2 shows a typical NNE, where separate
neural networks are formed for each of the colours from
Red to Grey and the content types ranging from very low to
very high. Here, we introduce parallel learning of neural
network tree ensembles with MapReduce.
Figure 2. Neural Network Ensemble for Image Classification
C. MapReduce Neural Network Implementation
The parallel framework offered by MapReduce is highly
suitable for the neural network ensemble technique of our
proposed CBIR framework. It can perform efficient data-
intensive computations and machine learning for image
classification and retrieval from very large digital image
collections. We describe the implementation details of
MapReduce in this section.
MapReduce uses two functions called Map and Reduce
that process list of pairs <key, value>. The Map function
inputs a list input key and associated values and produces a
list of intermediate <key, value> pairs. For CBIR, the image
features represent the keys and image files as values. Next,
grouping and shuffling of intermediate pairs with same keys
are performed. The Reduce function then does the merge
operation on all intermediate pairs for the same key and
outputs the results. Here the Reduce function can be
2012 12th International Conference on Hybrid Intelligent Systems (HIS) 65
executed in parallel. An illustration of the MapReduce
implementation for our proposed CBIR framework is shown
in Figure 3. For images, the process is illustrated as follows
The neural network ensemble is applied into a two-stage
MapReduce process, with one for training the classifier and
the other for validating the classifier. All the input pairs,
outputs pairs and intermediate pairs are stored in the Hadoop
Distributed File System (HDFS) as it provides a large-scale
data storage infrastructure based on clusters in the cloud. As
shown in Figure 1, the overall scheduling and running of the
MapReduce jobs are managed by the Hadoop with a job
tracker in a master node that assigns tasks to slave node and
each task would consist of multiple Map and Reduce
functions. It takes care of balancing the tasks and optimising
the overall runtime, making our proposed CBIR real-time
efficient for processing very large image datasets.
Figure 3. Illustration of MapReduce Implementation for CBIR
IV. E
XPERIMENTAL RESULTS AND FUTURE WORK
The experiment conducted consists of five main stages
and a preliminary testing of our proposed CBIR technique
was conducted using an image dataset of about thousand
images collected from public domains. We included images
of different categories such that different colours and hues
were covered to a great extent and all the content types were
included in the evaluation. Some of the categories used for
testing included images of babies, beaches, birds, boats,
cars, dogs, fireworks, flowers, landmarks, nature, planes,
planets, sunsets, waterfalls and weddings, etc. The following
five stages were performed in our CBIR .
Stage 1: Feature Extraction - Here the RGB and HSV values
were combined to determine colours (Red, Orange, Yellow,
Green, Cyan, Blue, Purple, Magenta, Pink, Black, White
and Grey). Colour pixels were extracted and for each pixel
the colour ranges were determined. We calculated the colour
content by incrementing the relevant colour counter when
the relevant colour pixel was found and the total for the
entire colours were also calculated to estimate the
percentage of each colour.
Stage 2: Feature Storage using MapReduce - Features
extracted in Stage 1 were stored in text files for each image.
These are in the form of pairs <Feature#, File#> of the Map
function of the MapReduce framework.
Stage 3: Natural Language Query - The user query could
have a combination of colours from the set {red, orange,
yellow, green, cyan, blue, purple, magenta, pink} and
content type in natural language terms from the set {no
colour, very low, low, medium, high, very high}. This way
user need not pose a similar image for the query. An
example output of the features of the experiment with
natural language query, where we can see the main colours
as magenta, pink, purple and black is shown in Figure 4.
Image
Colour %
Fuzzy Logic
Terms
Red 0.35
No Colour
Orange 0.02
No Colour
Yellow 0.01
No Colour
Green 0.01
No Colour
Cyan 0.01
No Colour
Blue 1.16
Very Low
Purple 36.93
Medium
Magenta 24.38
Low
Pink 29.06
Medium
Black 5.4
Very Low
White 2.65
Very Low
Grey 0.04
No Colour
Figure 4. Features extracted using natural language query
Stage 4: Image Classification using MapReduce - In this
stage the image is classified based on the fuzzy logic
membership function. We used the following fuzzy classes
for colour contents for each image: very low [0.05, 0.1], low
[0.11, 0.35], medium [0.36, 0.65], high [0.66, 0.80], very
high [0.81, 1.0]. A neural network ensemble (NNE) based
fusion of classes are done and the image classification is
done using the Reduce function of the MapReduce
framework.
66 2012 12th International Conference on Hybrid Intelligent Systems (HIS)