HAL Id: hal-01096624
https://hal.inria.fr/hal-01096624
Submitted on 17 Dec 2014
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Detection and Inpainting of Facial Wrinkles Using
Texture Orientation Fields and Markov Random Field
Modeling
Nazre Batool, Rama Chellappa
To cite this version:
Nazre Batool, Rama Chellappa. Detection and Inpainting of Facial Wrinkles Using Texture Orienta-
tion Fields and Markov Random Field Modeling. IEEE Transactions on Image Processing, Institute
of Electrical and Electronics Engineers, 2014, 23, pp.3773 - 3788. �10.1109/TIP.2014.2332401�. �hal-
01096624�
1
Detection and Inpainting of Facial Wrinkles using
Texture Orientation Fields and Markov Random
Field Modeling
Nazre Batool, Member, IEEE, and Rama Chellappa, Life Fellow, IEEE
Abstract—Facial retouching is widely used in media and
entertainment industry. Professional software usually require
a minimum level of user expertise to achieve the desirable
results. In this paper, we present an algorithm to detect facial
wrinkles/imperfection. We believe that any such algorithm would
be amenable to facial retouching applications. The detection
of wrinkles/imperfections can allow these skin features to be
processed differently than the surrounding skin without much
user interaction. For detection, Gabor filter responses along
with texture orientation field are used as image features. A
bimodal Gaussian mixture model (GMM) represents distribu-
tions of Gabor features of normal skin vs. skin imperfec-
tions. Then a Markov random field model (MRF) is used to
incorporate the spatial relationships among neighboring pix-
els for their GMM distributions and texture orientations. An
Expectation-Maximization (EM) algorithm then classifies skin
vs. skin wrinkles/imperfections. Once detected automatically,
wrinkles/imperfections are removed completely instead of being
blended or blurred. We propose an exemplar-based constrained
texture synthesis algorithm to inpaint irregularly shaped gaps left
by the removal of detected wrinkles/imperfections. We present
results conducted on images downloaded from the Internet to
show the efficacy of our algorithms.
Index Terms—Facial Wrinkles, Skin Imperfections, Markov
Random Field, Gaussian Mixture Model, Gabor Features, Tex-
ture Orientation Fields.
I. INTRODUCTION
D
IGITAL image inpainting refers to the filling of the gaps
of arbitrary shapes in an image so that they seem to
be parts of the original image. Several applications of digital
inpainting have been reported in the last decade e.g. filling
occlusions/gaps, removal of objects, image reconstruction by
removing scratches or other degradation [1], [2], [3], [4],
[5], [6], [7], [8], [9], [10], [11], [12], [13]. Here we propose a
specific application of digital inpainting to remove facial wrin-
kles and imperfections. Traditionally, beautification of skin or
facial re-touching in images has been done by professionals
using high-end software e.g. Adobe Photoshop
TM
. Several user
friendly smart phone applications (e.g. Visage Lab
TM
[14],
N. Batool is currently with the AYIN team at INRIA-Sophia Antipolis, 2004
route des Lucioles, 06902 Sophia Antipolis Cedex, France. R. Chellappa is
with the Department Of Electrical and Computer Engineering and the Center
for Automation Research, UMIACS, University of Maryland, College Park,
MD 20742, USA.
E-mail: nazr.e.batool@gmail.com, rama@umiacs.umd.edu
The first author would like to acknowledge the support of Ful-
bright/HEC(Pakistan)/USAID PhD Scholarship. However, the contents of this
publication have not been approved by the representing agencies/governments
of the USA/Pakistan.
Beautify
TM
[15], Perfect365
TM
[16]) which provide minimum
user interaction for facial touch ups have also been introduced.
However, both professional and user-friendly software have
limitations. Professional software require significant user in-
teractions where results are subjective, depending on user’s
expertise. Whereas user-friendly applications developed for
smart phones, while performing an overall beautification or
making up of skin with minimum user interaction, do not target
specific skin imperfections e.g. deep wrinkles, acne, scars etc.
An example is shown Fig. 1, where overall beautification of
skin fades wrinkles and moles but does not remove them
completely. The reason may be that these applications seem
to process all the skin regions equally and do not make
distinction between skin vs. skin imperfections. The results
can be improved if skin imperfections are detected as a
pre-processing step and then processed differently from the
surrounding skin.
The current state-of-the-art approach for the removal of
wrinkles is an image painting algorithm proposed by Georgiev
[17]. The algorithm is based on the widely used Poisson
image editing tool [18] and provides improved seamless
image cloning through better handling of lighting variations.
The algorithm works behind the Healing Tool in Adobe
Photoshop
TM
. Image painting is slightly a different applica-
tion from image inpainting. The former deals with inclusion
(painting) of a smaller image region in a larger image where
both source and destination image regions are provided by
the user. The latter deals with the automatic filling of a
gap/occlusion, mostly provided by the user, in an image based
on local and/or global image characteristics and does not
require a source image. However, both applications share the
requirement of seamless boundaries. Our work is closer to
image inpainting than image painting because both source and
destination image areas are selected automatically. We make
the following observations about the current facial retouching
software as a motivation for our proposed work.
1) Significant user interaction is required with the Adobe
Healing Tool for the selection of source and destination
skin patches resulting in subjective results depending on
user expertise.
2) In the case of more user-friendly applications, facial
retouching results in the so-called flawless skin. The
processing of skin in an image smoothes wrinkles and
skin imperfections but does not remove them completely.
3) Regarding image inpainting techniques, both structure
2
Fig. 1. Typical results of facial retouching for a smart phone application
[14]. (a) Original Image. (b) Image after retouching. Note that wrinkles on
forehead and brown spots on cheeks are deemphasized due to blending but
still visible.
and texture inpainting techniques are not applicable
directly to the skin. Wrinkles and skin imperfections do
not appear as edges/boundaries and, hence, structural
inpainting is not appropriate. Also, as wrinkles are not
homogeneous texture patterns, texture inpainting is not
effective.
The main contributions of this paper are as follows:
1) An algorithm based on the fusion of Gabor features and
texture orientation fields in the framework of Markov
field modeling (MRF) is proposed to detect wrinkles
and other imperfections in the surrounding skin.
2) A variation of exemplar-based texture synthesis is pro-
posed to fill the gaps of irregular shapes.
3) Both detection and inpainting of wrinkles are unsuper-
vised with minimum user interaction thus minimizing
the subjectivity introduced by the user’s expertise.
4) No ‘retouching’ or ‘beautification’ of the rest of
the facial skin is done while inpainting skin wrin-
kles/imperfections.
The organization of this paper is as follows. In section II, we
present an overview of some related work. In section III, we
present the details of our detection and inpainting algorithms.
Experiments and discussion are presented in section IV. Fi-
nally, we conclude the paper in section V.
II. R
ELATED WORK
Image inpainting methods target one or both of the structure
and texture of an image. The difference between the image
attributes of structure and texture of an image requires different
inpainting methods. A detailed survey of image inpainting
methods can be found in [13], [10], [12]. Most texture in-
painting methods require user input or some masking function
to highlight the gap/occlusion to be filled (e.g. the work
by Criminisi [6]). Some examples of automatic filling of
scratches, rectangular blocks or random noise can be found in
[13], [1], [11]. Shi and Chang introduced a patch-based multi-
resolution/multi-layer approach to restore the paintings dam-
aged by red scratches [13]. Their approach involved a mecha-
nism to detect the damaged areas first where the variance in the
color of a patch at a specific resolution was used to determine
if a patch had damaged pixels. In contrast, the inpainting
methods in [11], [1] do not consider any explicit detection of
gaps to be filled. These techniques are based on the analysis of
different layers containing low vs. high frequency details. The
low frequency layer determines the piecewise smooth regions
of the image and the high frequency layer determines the
texture. The recovery of these layers automatically fills the
gaps without their being detected explicitly. In case where
more than one texture is surrounding the gap, sophisticated
techniques are used for combining different textures [8], [6].
Once a suitable combination of different textures has been
found, the gap is filled by existing texture synthesis techniques.
For example, Grossauer [8] used the exemplar-based texture
synthesis technique given in [19] and Criminisi et al. [6] used
a synthesis method similar to [20].
The specific application of wrinkle removal is different as
wrinkles are not artifacts or separate objects to be removed.
Wrinkles are an inherent part of the skin and are visible
only due to their discontinuous nature in surrounding skin
texture. Recently, the detection of wrinkles as sequences of
line segments/curves was reported by Batool and Chellappa
([21]). However, this method is not applicable here because
of two reasons. First, wrinkles are localized as curves and the
surrounding folds of skin due to a wrinkle are not detected.
Second, the method reported in [21] is based on line segments
and cannot be used to detect other oval like skin imperfections.
Our wrinkle inpainting approach is based on Poisson editing
and a variation of exemplar-based texture synthesis. However,
we use a novel approach to detect wrinkles and skin imper-
fections. In the following section we present our approach in
detail.
III. A
PPROACH
An image inpainting technique for textures has three main
steps, (a) finding a suitable texture template in the image to fill
in the gap with, (b) calculating the seamless warping between
the template and the gap and (c) filling the gap via texture syn-
thesis. Since we are proposing unsupervised image inpainting,
an additional step is required to detect wrinkles automatically.
The process of wrinkling creates deep creases and causes
curvature in the surrounding skin. The resulting skin curvature
causes specific intensity gradients in skin images which look
like discontinuities in surrounding skin textures. An accurate
inpainting of wrinkles will require both the wrinkle crease and
the surrounding curved skin to be removed. In section III-A,
we present our approach for detection. Regarding step (a)
of image inpainting mentioned above, we select skin patches
surrounding the detected wrinkles. This is due to the fact that
the skin texture can vary significantly within a small region of
face. The skin patches closest to the wrinkles have the most
similar looking skin texture. Regarding steps (b) and (c), we
use an exemplar-based texture synthesis method based on the
work of Efros and Freeman [22]. The details of our texture
synthesis method will be presented section III-B.
A. Automatic Detection of Regions with Wrinkles
We use texture orientation field proposed by Rao and
Schunk [23] and Gabor filter responses as image features.
3
Fig. 2. Image features used for segmentation. (a) Forehead image in gray scale. (b) Maximum Gabor amplitude response (values [4.8, 132] scaled to the
gray scale values [0,255]. (c) Texture orientation field.
The orientation field highlights the discontinuities in the
normal flow of skin texture whereas the Gabor filter responses
highlight the intensity gradients in any directions. The two
types of features are fused using Gaussian Mixture Models
(GMM) and Markov random field representation. The GMM
classifies filter responses as a bimodal distribution for skin
vs. skin imperfections. The MRF representation allows us to
incorporate spatial relationship among GMM distributions of
neighboring pixels and to fuse the orientation fields to reshape
the class probabilities.
1) Computation of Orientation Fields using Gabor Filters:
Several oriented feature detectors have been developed includ-
ing steerable Gaussian second-derivative filters, line operators
and Gabor filters. A comparative study can be found in [24]
where the real Gabor filters were assessed to be the best
detector of oriented features. The real Gabor filter kernel is
given by
g(x
1
,x
2
)=
1
2πσ
x
1
σ
x
2
exp
−1
2
x
′
2
1
σ
2
x
1
+
x
′
2
2
σ
2
x
2
cos(2πfx
′
1
)
(1)
where
x
′
1
x
′
2
=
cos α sin α
− sin α cos α
x
1
x
2
(2)
The parameters α and f denote the orientation angle and
frequency of the sinusoidal factor respectively and {σ
x
1
,σ
x
2
}
denote the standard deviations of the Gaussian envelope in 2D
plane. Let {g
k
(x
1
,x
2
),k =0, ··· ,K − 1} denote the set of
real Gabor filters oriented at angles α
k
= −
π
2
+
πk
K
where K
is the total number of equally spaced filters over the angular
range
−π
2
,
π
2
. Let {I(x
1
,x
2
); x
1
=1...N
1
,x
2
=1...N
2
}
denote the input image in gray scale and I
f
k
(x
1
,x
2
) denote
the image filtered by the filter g
k
(x
1
,x
2
). Then the orientation
field, θ
I
(x
1
,x
2
) for the image is computed as follows:
θ
I
(x
1
,x
2
) = arg max
k
I
f
k
(x
1
,x
2
) (3)
i.e. at every pixel, the orientation field is equal to the ori-
entation angle of the filter resulting in the maximum filtered
response at that pixel. The corresponding maximum amplitude
among the filtered responses is given as:
I
′
(x
1
,x
2
)=max
k
I
f
k
(x
1
,x
2
) (4)
The set of the maximum filter response and the orientation
angle at every pixel, {I
′
(x
1
,x
2
),θ
I
(x
1
,x
2
)}, constitutes im-
age features for automatic detection of wrinkle regions. Fig. 2
shows a forehead image with the corresponding maximum
responses and the orientation field. The orientation angle is
calculated at every pixel, however, the orientation field in
Fig. 2(c) is drawn by placing needles at every 3rd pixel. Every
needle is of length of 3 pixels and is placed in the direction
of the orientation angle.
At high resolution, skin texture appears to be granular
resulting in random orientation angles. However, the skin
creases of wrinkles and the skin pigments related to other
imperfections (e.g. brown spot, moles) smooth out the granular
skin texture. As a result, the orientation field depicts two
significant properties in wrinkled regions, (a) a dominant angle
of zero degrees and (b) pixels with zero orientation angle
appear in clusters. Fig. 3 depicts these two properties of
orientation field due to wrinkles. We exploit these observations
to formulate the GMM-MRF model based, two-class labeling
of images into wrinkles and non-wrinkle regions. The next
section describes the model in detail.
Fig. 3. (Right) Rectangle ’A‘ shows the skin texture used as template whereas
rectangle ’B‘ shows change in skin texture due to a wrinkle. (Left) Orientation
field at high resolution, note that sites corresponding to the wrinkle have
orientation angle of zero degrees.
2) Gaussian Mixture Model based on Markov Random
Field (GMM-MRF): The motivation behind using the GMM-
MRF model is the fact that the Gabor filter responses or the
texture orientation field, when used exclusively, are important
but insufficient features to detect the wrinkled regions. For
example, Fig. 4(a) shows the result of thresholding Gabor
amplitude responses in the range [4.8, 132] with the threshold
value of 35 and Fig. 4(b) shows image sites with orientation
angles lying in the range [−5, +5]. Fig. 4(c) shows the product
of both results and resembles more closely to the actual
wrinkles by reducing false positive in either of the Fig. 4(a)
and Fig. 4(b). We make the following observations to justify
GMM-MRF modeling.
GMM:Histograms of Gabor response amplitude I
′
(x
1
,x
2
)
typically follow the Beta distribution with heavy
tails. For example Fig. 5 shows a histogram of the
Gabor amplitude response for the image in Fig. 2(a).
An intelligent thresholding of the Beta distribution
can provide a good starting point for any segmenta-
4
Fig. 4. Results of thresholding. (a) Thresholding maximum Gabor amplitude at value 35. (b) Thresholding orientation field at absolute angle values of less
than 5 degrees. (c) Product of images in (a) and (b).
Fig. 5. Histogram of the Gabor features in Fig. 2(b).
tion technique. Modeling of Gabor responses as Beta
distribution may seem an obvious choice. However,
we take the simpler approach of Gaussian mixture
models for its more developed theory. A similar
approach can be found in [25] where the authors
used the GMM to model the Beta distribution for
segmentation of SAR images.
MRF: Since class labels do not depend solely on the Gabor
response amplitude, a simple thresholding of Beta
distribution does not work. There is always some
under segmentation or over segmentation present.
Texture orientation field has to be incorporated to
aid thresholding by reshaping the probability of each
class. An MRF framework enables not only the incor-
poration of spatial dependencies among neighboring
pixels but also the fusion of texture orientation fields
and Gabor amplitude responses.
We first present the GMM and MRF models and then discuss
how the MRF model is used to fuse the orientation field with
Gabor amplitude responses.
Under GMM, the density function for the observation at
pixel (x
1
,x
2
) is given as:
f(I
′
(x
1
,x
2
)|Π, Θ) =
J
j=1
π
j
x
1
,x
2
Φ(I
′
(x
1
,x
2
)|µ
j
,σ
j
) (5)
where Φ(I
′
(x
1
,x
2
)|µ
j
,σ
j
) is the standard Gaussian distri-
bution with mean µ
j
and standard deviation σ
j
and Θ=
{(µ
j
,σ
j
); j =1, ··· ,J} is the parameter set of Gaussian
mixture distributions. The set of mixing proportions, Π=
{π
j
x
1
,x
2
; x
1
=1, ··· ,N
1
; x
2
=1, ··· ,N
2
; j =1, ··· ,J}
satisfies the following constraints:
0 ≤ π
j
x
1
,x
2
≤ 1 and
j
π
j
x
1
,x
2
=1 (6)
Let I
′
denote the ensemble of random variables I
′
(x
1
,x
2
) as
follows:
I
′
= {I
′
(x
1
,x
2
); x
1
=1, ··· ,N
2
; x
2
=1, ··· ,N
2
} (7)
Then, assuming statistical independence of individual pixel
sites, the joint conditional density function of the whole image
can be written as:
p(I
′
|Π, Θ) =
N
1
x
1
=1
N
2
x
2
=1
J
j=1
π
j
x
1
,x
2
Φ(I
′
(x
1
,x
2
)|µ
j
,σ
j
) (8)
According to Bayes’ theorem, the posterior probability can be
written as follows:
p(Π|I
′
, Θ) ∝ p(I
′
|Π, Θ) × p(Π) (9)
Gaussian mixture models based on MRF (GMM-MRF) are
proposed to impose spatial smoothness constraints between
neighboring pixels [26]. Under MRF models , the prior distri-
bution of the mixing proportion of a pixel (x
1
,x
2
), denoted by
π
j
x
1
,x
2
, depends on those of its neighboring pixels. The prior
joint distribution of π
j
x
1
,x
2
for all pixels is given by the Gibbs
distribution:
p(Π) =
1
Z
exp(−
U(Π)
T
) (10)
where Z is the normalization constant, U(Π) is the Gibbs
energy function and T is a constant called temperature. Ac-
cording to (8), (9) and (10), the posteriori log-density function
can be derived as:
L(Π|I
′
, Θ) = log p(Π|I
′
, Θ)
=
N
1
x
1
=1
N
2
x
2
=1
log
⎧
⎨
⎩
J
j=1
π
j
x
1
,x
2
Φ(I
′
(x
1
,x
2
)|µ
j
,σ
j
)
⎫
⎬
⎭
+logp(Π)
=
N
1
x
1
=1
N
2
x
2
=1
log
⎧
⎨
⎩
J
j=1
π
j
x
1
,x
2
Φ(I
′
(x
1
,x
2
)|µ
j
,σ
j
)
⎫
⎬
⎭
− log Z −
U(Π)
T
(11)
The expectation maximization (EM) algorithm is usually used
to estimate the parameters of a GMM distribution. However,
the inclusion of prior distribution to GMM via an MRF
introduces additional complexity and the M-step of the EM
algorithm cannot be directly applied to estimate the model
parameters from the observations. Various approximations
have been introduced to tackle this problem. Recently, Nguyen
et al.[26] introduced a novel way of incorporating spatial
correlations in MRF model which allows a close form solution
![Fig. 6. Binary labeling results. (a) Initial labels obtained after using GMM functions in MatlabTM . (b) labeling results after 15 iterations of our GMMMRF method (c) labeling results using GMM-MRF method in [26] (d) Resulting gaps in the skin image.](/figures/fig-6-binary-labeling-results-a-initial-labels-obtained-25zdorpn.webp)
