6SDWLDO'RPDLQ'HFLVLRQ%DVHG,PDJH)XVLRQ8VLQJ
6XSHULPSRVLWLRQ
Maitreyi Abhyankar, Arti Khaparde
Department of E&TC
Maharashtra Institute of Technology, SPPU
Pune, India
arti.khaparde@mitpune.edu.in
maitreyi_abhyankar@yahoo.com
Vaidehi Deshmukh
Department of E&TC
RMD College of Engineering, SPPU
Pune, India.
vaidehideshmukh@sinhgad.edu
Abstract—
Image fusion is the process to combine two or more
images such that the enhanced output image contains all the
relevant information. This paper aims to evaluate algorithms that
fuses multi-focus images. With an aim to enhance the sharpness of
the final fused image and also to reduce the computational
complexity, a novel method is proposed that uses the Sobel
operator. This method is evaluated using mean squared error,
peak signal to noise ratio, structural similarity index, entropy,
mutual information, image quality index and run time. The
optimum weights required to fuse two images are calculated from
image statistics using genetic algorithm (GA). The results show
that this superposition method performs at par with discrete
wavelet transform (DWT) with reduction in run time.
Keywords—Image fusion, superimposed, Sobel operator, GA
I. I
NTRODUCTION
Image fusion forms a new image that is more informative than
its constituent images. Due to non-uniform distance of objects
from camera and limited depth-of-focus of the camera lens, not
all the objects can be focused upon simultaneously. However, it
is possible to obtain a number of images, each with a different
object in focus, which can be combined together using image
fusion techniques to get a final image with all the scene objects
in focus [1].
The fusion of images can take place at two different levels-
pixel based low level and decision based high level [2]. The
pixel based techniques act upon the corresponding pixels of the
input images. On the other hand, the decision based techniques
take into account various features of the images such as edges or
regions. These techniques have the advantage of lesser noise
sensitivity and better contrast compared to the pixel based
techniques, at the cost of complexity.
Image fusion techniques can also be broadly classified into
two categories-spatial domain techniques and transform domain
techniques [3]. Spatial domain methods directly work on the
pixel value, whereas transform domain techniques include
transforming the images using mathematical equations.
A simple spatial domain technique results in images that
have poor contrast and sharpness. Another pixel level image
fusion method involves giving the entire image as an input to the
genetic search [7]. However, this method is computationally
very expensive.
There are pyramid based methods, whose basic principle is
to decompose the original image into pieces of sub-images with
different spatial resolutions through some mathematical
operations. The Laplacian pyramid is derived from the
Gaussian pyramid, which is a multi-scale representation
obtained through a recursive low-pass filtering and decimation.
So, the Laplacian pyramid decomposition is divided into two
steps: the first is Gaussian pyramid decomposition, the second
is from Gaussian pyramid to Laplacian pyramid [5]. Thus such
method uses the second order derivatives, which are more
immune to noise.
However, the proposed novel technique used the first order
derivative without decomposing the original image into sub-
images with different spatial resolutions. Thus reducing the
mathematical complexity. It uses two steps to fuse an image-
firstly it finds the edges using edge detection method and then
they are given as input to genetic algorithm for calculating the
weights in which the two images should be fused.
The remainder of the paper is organised as follows: Section
II describes the proposed algorithm. Section III gives the
parameters on which the evaluation were done. Section IV deals
with the results and Section V is conclusion..
II. P
ROPOSED
M
ETHOD
A. Superimposing Sobel detected edges
Focus and sharpness are features closely related to the quality
of the image. Sharper images have more information than the
blurred ones. It is often observed that fused images have blurred
edges. Thus with an aim to improve the edge sharpness of the
fused images, this algorithm superimposes the edges on the
input images.
The Sobel operator uses different kernels to detect edges along
x and y directions. The kernels are designed to respond
maximally to the edges that are vertical and horizontal, relative
to the pixel grid. The Sobel operator is a combination of
differentiation and Gaussian smoothing, thus making it more
resistant to noise, compared to the Prewitt operator. It not only
gives an estimate of the direction of the edge, but also gives the
978-1-5090-0806-3/16/$31.00 copyright 2016 IEEE
ICIS 2016, June 26-29, 2016, Okayama, Japan
magnitude of the edge at a particular point. Canny edge detector
requires careful selection of the two thresholds without which
the edges are not detected properly. Thus Sobel operator was
chosen to give good results avoiding the critical selection of the
threshold values.
The algorithm proceeds by detecting the edges for the input
images. The kernel used to detect edges along x direction is:
െͳͲͳ
െʹͲʹ
െͳͲͳ
൩ (1)
The edges along y direction are detected using:
െͳ െʹ െͳ
ͲͲͲ
ͳʹͳ
൩ (2)
The edges are then averaged to form a new edge image E. This
image is superimposed on the first input using weighted
addition method.
Image3 = k1*Image1 + k2*E (3)
݇
ଵ
ൌ
௩௫௩௨ூ
భ
ெ௫௫௩௨ூ
భ
(4)
݇
ଶ
ൌ
ሺ
ͳെ݇
ଵ
ሻ
*0.1 (5)
The intermediate image Image3 is superimposed on the second
input image to form the final output image.
ܨ݈݅݊ܽൌ݇
ଷ
כܫ݉ܽ݃݁
ଶ
݇
ସ
כܫ݉ܽ݃݁
ଷ
(6)
݇
ଷ
ൌͳെሺ
௩௫௩௨ூ
మ
ெ௫௫௩௨ூ
మ
ሻ
(7)
݇
ସ
= 1 (as it is used for the intermediate image that is
already superimposed) (8)
B. Feature vector for Sobel based GA
For Sobel based genetic algorithm, the Sobel operator is applied
along x and y directions forming two edge images. Normalised
central moments up to order 3 for each edge image are
calculated-forming a total of 16 moments. Mean and standard
deviation of edges along x and y directions are calculated.
Finally, the feature vector of size 1 X 20 is formed by combining
the moments, mean and standard deviation
.
C. Genetic algorithm based on edges detected by Sobel
operator
The genetic search [7] implemented starts by generating a
population of size 100. This population consists of randomly
generated binary strings of length 10. Before evaluating the
individuals, they are converted from binary to decimal
equivalent such that every binary string lies in the range of 0 to
This conversion is given as:
L
l
+ (U
l
– L
l
) * Decimal equivalent / (
ʹ
െͳ
) (9)
Where, Lower limit (L
l
)=0, Upper limit (U
l
) =1, n = Number
of bits in the binary.
The individuals are evaluated using mean squared error as the
fitness function. Mathematically, this fitness function is given
as:
Fitness=
ቄ
ଵ
ଶ
ቀ
ଵ
ௗ
σ
ሾ
ܨ
௧
ሺ݅ሻെܫ݉݃
ଵ
ሺ
݅
ሻ
ሿ
ଶ
ௗ
ୀଵ
ቁቀ
ଵ
ௗ
σ
ሾ
ܨ
௧
ሺ
݅
ሻ
െܫ݉݃
ଶ
ሺ݅ሻ
ሿ
ଶ
ௗ
ୀଵ
ቁቅ
(10)
d is the dimension of the input vector which is 1 x 40, calculated
as described in section II B.
ܨ
௧
( i ) is the intermediate fused
image obtained by using the equation:
ܨ
௧
ሺ
݅
ሻ
ൌܹ
௧ଵ
כܫ݉݃
ଵ
ܹ
௧ଶ
כܫ݉݃
ଶ
(11)
ܫ݉݃
ଵ
and ܫ݉݃
ଶ
are the input images, ܹ
௧ଵ
is the weight
individual being evaluated and:
ܹ
௧ଶ
= (1-ܹ
௧ଵ
) (12)
The value of this fitness function should be as small as possible.
However, it can also be transformed to make this fitness function
a maximisation fitness using the function:
F(x) = 1/ (1 + Fitness) (13)
Selection is done by considering the individuals that have the
highest fitness values. Different methods exist for the selection
process such as- Roulette wheel selection, Rank based selection,
Tournament selection etc. [7] uses tournament based selection
method with a tournament size of 2. The next step is performing
crossover and mutation operations on the selected individuals. It
uses a crossover probability of 0.7 and a mutation probability of
0.002. Uniform crossover and bit-flip mutation is used. These
modified individuals replace the existing population. The
process is repeated for 25 generations. It is experimentally
observed that this search is converging in 20-23 generations.
After the last generation, the optimum candidate is selected. The
second weight is obtained as (1-
ܹ
ଵ
). On obtaining the two
weights, each is assigned to the input images. Final fused image
is obtained as a result of the addition of these optimally weighted
images as:
ܨ
ൌܹ
ଵ
כܫ݉݃
ଵ
ܹ
ଶ
כܫ݉݃
ଶ
(14)
The block diagram for the process is described in Figure 1:
Fig. 1 Flow Diagram for Proposed method
III. E
VALUATION OF THE FUSED IMAGES
:
The basic requirement is that the output images should contain
all the valid and useful information present in the source images
without introducing any form of distortion. The resultant
images can be compared visually, but, these methods, though
very powerful, are subjective in nature. Hence, objective
statistical methods are used [1-9].
1) Root Mean Squared Error:
RMSE=
ට
ଵ
ሾ
σσ
ܫ݉݃ͳ
ሺ
݅ǡ݆
ሻ
െܫ݉݃ʹሺ݅ǡ݆ሻሿ
ିଵ
ୀ
ିଵ
ୀ
ଶ
మ
(15)
m, n: size of the input images Img1 and Img2
2) Peak Signal to Noise Ratio:
Peak signal to noise ratio gives the ratio of the maximum power
of a signal and the power of corrupting noise.
PSNR = 20 log10 (
ெ௫
ξெௌா
) (16)
Max: Maximum possible pixel value, as 8 bits are used to
represent a pixel, Max=255
3) Structural similarity index:
SSIM (x, y) =
ሺଶఓ
ೣ
ఓ
ା
భ
ሻሺଶఙ
ೣ
ା
మ
ሻ
ሺఓ
ೣ
మ
ାఓ
మ
ା
భ
ሻሺఙ
ೣ
మ
ାఙ
మ
ା
మ
ሻ
(17)
Where μ
x
the average of x; μ
y
the average of y;
ߪ
௫
ଶ
the variance of x;
ߪ
௬
ଶ
the variance of y;
σ
xy
the covariance of x and y;
ܿͳൌ
ሺ
݇
ଵ
ܮ
ሻ
ଶ
, ܿʹൌ
ሺ
݇
ଶ
ܮ
ሻ
ଶ
(18)
L is the dynamic range of the pixel-values;
݇
ଵ
ൌͲǤͲͳ
and
݇
ଶ
ൌͲǤͲ͵
by default.
SSIM is used to model any image distortion as a combination
of correlation losses, radiometric and contrast distortions.
Higher the value of SSIM, more similar are the images. It can
have a maximum possible value of 1.
4) Entropy:
Entropy is used to give the amount of information contained in
an image. Mathematically it is given as:
E =
െ
σ
ܲ
݈݃
ଶ
ܲ
ିଵ
ୀ
(19)
P
i
is the probability of occurrence of a pixel value in the image,
L is the number of intensity levels in the image. Increase in the
value of entropy after fusion indicates that the information
contained in the image has increased.
5) Mutual Information:
It is an indicator of the information obtained from the source
images and the quantity that is conveyed by the fused image.
MI =
ܯܫ
ிǡூଵ
ܯܫ
ிǡூଶ
(20)
ܯܫ
ிǡூଵ
=
σ
ܲ
ிǡூଵ
ሺ݂ǡ݅ͳሻ
ଶ
ಷǡభ
ሺǡଵሻ
ಷ
ሺሻ
భ
ሺଵሻ
(21)
ܯܫ
ிǡூଶ
=
σ
ܲ
ிǡூଶ
ሺ݂ǡ݅ʹሻ
ଶ
ಷǡమ
ሺǡଶሻ
ಷ
ሺሻ
భ
ሺଶሻ
(22)
Where
ܯܫ
ிǡூଵ
denotes the mutual information between the
fused image and the first input image,
ܯܫ
ிǡூଶ
denotes the
mutual information between the second input and the fused
image,
ܲ
ிǡூଵ
ሺ݂ǡ݅ͳሻ
and
ܲ
ிǡூଶ
ሺ݂ǡ݅ʹሻ
are the joint
histograms of fused image, input 1and fused image, input 2
respectively,
ܲ
ி
ሺ
݂
ሻ
ǡܲ
ூଵ
ሺ݅ͳ
) and
ܲ
ூଵ
ሺ
݅ʹ
ሻ
are the histograms
of the fused image, input 1 and input 2 respectively.
6) Run Time:
It gives a measure of the time required for complete run of the
algorithm in seconds.
7) Image Quality Index [9]:
Mathematically given as:
ସఙ
ೣ
௫
ҧ
௬
ത
൫ఙ
ೣ
మ
ାఙ
మ
൯
ሾ
ሺ
௫
ҧ
ሻ
మ
ା
ሺ
௬
ത
ሻ
మ
ሿ
(23)
where
ݔҧǡݕത
are the means of x and y respectively,
ߪ
௫௬
is the
covariance of x and y,
ߪ
௫
ଶ
ǡߪ
௬
ଶ
are the variances of x and y
respectively.
IV. R
ESULTS
The algorithms : - Image Fusion Using Genetic Algorithm and
Discrete Wavelet Transform proposed [6] and Genetic
algorithm based on edges detected by Sobel operator were both
implemented and tested on different types of images from a
database of multi-focus images and also on real time images
captured by a camera, focusing on different object in every
image. The analysis of the result were done on the different
parameters as explained in the pervious section. Here results for
6 images in each categoryy are shown for reference. For
subjective analysis 50 subjects were considered. The subjective
and objective analysis for the image from the database and for
real time image caputered from camera is given in the Section
IV. A and IV.B respectively.
A. Results for database image
(a) (b)
( c) (d)
(e) (f)
Fig. 2 (a)-(b): Data base input images, (c) DWT (d)DWT_GA (e) SUP (f)
SUP_GA
TABLE I. COMPARISION RESULTS FOR DATABASE IMAGESET 1
Method
DWT DWT_GA SUP SUP-GA
Parameter
RMSE 12.29337633 12.29198814 12.46290628 12.29198814
PSNR 26.33740597 26.33934677 26.23599944 26.33934677
SSIM 0.813867487 0.813732134 0.81200342 0.813697015
ENTROPY 7.231954748 7.233046212 7.20405317 7.230991011
MI 4.517193245 4.542809862 4.520682512 4.541790674
IQI 0.789147424 0.788287993 0.781242432 0.788084485
TIME 0.107461748 0.774208658 0.005526063 0.641027041
TABLE II. (A)-(F) OBJECTIVE ANALYSIS OF DATABASE IMAGES
TABLE II. (A)
DB_ENTROPY
DWT DWT_GA SUP SUP-GA
Img 1 7.274102079 7.276063705 7.279682645 7.276063705
Img 2 7.124278087 7.122863208 7.125016254 7.122863208
Img 3 7.401747586 7.402183644 7.436018162 7.401685456
Img 4 7.2651839 7.26871736 7.280613934 7.26871736
Img 5 7.630098537 7.62981082 7.633059902 7.62981082
Img 6 7.456101171 7.455349685 7.472908158 7.457472519
TABLE II. (B)
DB_RMSE
DWT DWT_GA SUP SUP-GA
Img 1 1.91826451 1.91152555 1.94137463 1.91152555
Img 2 5.85085359 5.84905470 5.875658161 5.84905470
Img 3 4.22200801 4.21939606 5.378821475 4.21939606
Img 4 5.47229690 5.470035809 5.537205451 5.47003580
Img 5 8.22364723 8.221683078 8.279220459 8.2216830
Img 6 6.19894491 6.196928392 6.396782339 6.19692839
TABLE II. (C)
DB_PSNR
DWT DWT_GA SUP SUP-GA
Img 1 42.472634 42.533424 42.477594 42.53342446
Img 2 32.786419 32.791667 33.25994 32.79166767
Img 3 35.620422 35.631023 33.517562 35.63102308
Img 4 33.367410 33.374527 33.293689 33.37452747
Img 5 29.829514 29.833611 29.826373 29.83361158
Img 6 32.284448 32.290016 32.018395 32.29001684
TABLE II. (D)
DB_SSIM
DWT DWT_GA SUP SUP-GA
Img 1 0.982777041 0.982123634 0.981018979 0.982123634
Img 2 0.930889259 0.930452878 0.924458696 0.930452878
Img 3 0.956972799 0.956376947 0.955409133 0.956554622
Img 4 0.930143237 0.929757413 0.92827984 0.929798738
Img 5 0.87890549 0.87866285 0.876113805 0.878824499
Img 6 0.938319003 0.938011691 0.93627495 0.937855107
TABLE II. (E)
DB_MI
DWT DWT_GA SUP SUP-GA
Img 1 9.551883011 9.655036852 9.567111012 9.655036852
Img 2 6.4997739 6.561308933 6.686758519 6.561451091
Img 3 7.827594187 7.905515019 7.814607012 7.905515019
Img 4 6.555595817 6.614265014 6.564783308 6.614265014
Img 5 5.574346778 5.60367631 5.602074868 5.60367631
Img 6 6.881332977 6.93498181 6.872462701 6.934162586
TABLE II. (F)
DB_IQI
DWT DWT_GA SUP SUP-GA
Img 1 0.935682543 0.925304617 0.919558285 0.929710395
Img 2 0.873712795 0.87268237 0.861826782 0.87268237
Img 3 0.922875608 0.920257882 0.917886311 0.920339326
Img 4 0.900840182 0.898281339 0.896783403 0.898281339
Img 5 0.851772361 0.851094892 0.848525631 0.851094892
Img 6 0.876924905 0.876336084 0.873263578 0.873728318
Fig. 3 Comparison graph of run-time for database image sets
B. Results for real time images dataset
(a) (b)
(c) (d)
(e) (f)
Figure 4: (a)-(b)In-house input images, (c)DWT, (d)DWT_GA, (e)SUP,
(f)SUP_GA
TABLE III. COMPARISION RESULTS FOR REAL TIME IMAGE SET
Method
DWT DWT_GA SUP SUP-GA
Parameter
RMSE 16.34494177 16.34435492 18.35665383 16.34435492
PSNR 23.86342597 23.86403993 23.05998683 23.86403993
SSIM 0.842798432 0.842749184 0.839463641 0.842705444
ENTROPY 7.640887795 7.641077463 7.731927953 7.641090614
MI 5.301427461 5.30847386 5.332867629 5.308912345
IQI 0.653848904 0.653172121 0.647788754 0.653768346
TIME 0.292187607 0.978869865 0.031723194 0.700341361
TABLE IV (A)-(F) OBJECTIVE ANALYSIS OF REAL TIME IMAGES
TABLE IV (A)
TABLE IV (B)
RT_RMSE
DWT DWT_GA SUP SUP-GA
Img 1 20.6532295 20.65286544 21.42743903 20.65286544
Img 2 16.65166062 16.65131342 19.13375733 16.65131342
Img 3 15.28754074 15.28654024 15.30659951 15.28654024
Img 4 28.56503167 28.56498326 29.53373719 28.56498326
Img 5 32.20576404 32.20574529 34.09786658 32.20574529
Img 6 27.04314846 27.04312433 28.15428863 27.04312433
TABLE IV (C)
TABLE IV (D)
RT_SSIM
DWT DWT_GA SUP SUP-GA
Img 1 0.818907818 0.818984598 0.816233943 0.818984598
Img 2 0.849284887 0.849310637 0.844615324 0.849097836
Img 3 0.785957966 0.785965178 0.784197483 0.785965178
Img 4 0.762326912 0.762082945 0.760206148 0.762299459
Img 5 0.800947268 0.800945828 0.795289886 0.800945828
Img 6 0.821950081 0.821952094 0.816359774 0.821343789
TABLE IV (E)
RT_MI
DWT DWT_GA SUP SUP-GA
Img 1 4.873001463 4.876984936 4.845757594 4.880716815
RT_ENTROPY
DWT DWT_GA SUP SUP-GA
Img 1 7.579773661 7.580852115 7.498521817 7.58085211
Img 2 7.362242686 7.362771531 7.480970609 7.36169769
Img 3 7.261306599 7.261494279 7.243778203 7.26149427
Img 4 7.391961954 7.39242764 7.244967577 7.39177275
Img 5 7.438890956 7.439064454 7.212666618 7.43864487
Img 6 6.984488975 6.98400513 6.890249163 6.98400513
RT_PSNR
DWT DWT_GA SUP SUP-GA
Img 1 21.83124518 21.83154276 21.59659871 21.83154276
Img 2 23.70207574 23.70242563 22.57267635 23.70242563
Img 3 24.44412938 24.44525625 24.4737668 24.44525625
Img 4 19.01434398 19.01436827 18.85130113 19.01436827
Img 5 17.97233335 17.97233793 17.94376017 17.97331237
Img 6 19.48992689 19.48993356 19.16702031 19.48993356