State of the "Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video

TLDR: This paper presents a taxonomy of the 2D NPR algorithms developed over the past two decades, structured according to the design characteristics and behavior of each technique, and describes a chronology of development from the semiautomatic paint systems of the early nineties, through to the automated painterly rendering system of the late nineties driven by image gradient analysis.

Abstract: This paper surveys the field of nonphotorealistic rendering (NPR), focusing on techniques for transforming 2D input (images and video) into artistically stylized renderings. We first present a taxonomy of the 2D NPR algorithms developed over the past two decades, structured according to the design characteristics and behavior of each technique. We then describe a chronology of development from the semiautomatic paint systems of the early nineties, through to the automated painterly rendering systems of the late nineties driven by image gradient analysis. Two complementary trends in the NPR literature are then addressed, with reference to our taxonomy. First, the fusion of higher level computer vision and NPR, illustrating the trends toward scene analysis to drive artistic abstraction and diversity of style. Second, the evolution of local processing approaches toward edge-aware filtering for real-time stylization of images and video. The survey then concludes with a discussion of open challenges for 2D NPR identified in recent NPR symposia, including topics such as user and aesthetic evaluation.

Figures

Fig. 3. Adapting Haeberli’s framework [47] to randomly assign stroke size and order leads to loss of salient detail [48] and motivated later the use of image processing operators for stroke placement.

Fig. 8. Image analogies extend patch based texture in-filling techniques to match between a source image and its artistic rendering. Images are rendered in analogous styles using the learned mapping.

Fig. 11. Generalized pipeline for creating cartoon-like effects by local filtering [70], [82], [164]. After the conversion to CIELab, the input is iteratively abstracted using the bilateral filter. First, 1–2 bilateral filter iterations suppress noise, and outlines are extracted from the intermediate result using a DoG filter. Further iterations of the bilateral filter are performed, with luminance quantization applied afterwards. DoG edges and the output of the luminance quantization are then composited, followed by optional sharpening by warping and smoothing of the edges.

Fig. 4. Illustrating the coarse to fine rendering and curved path tracing components of Hertzmann’s painting algorithm [55].

Fig. 7. Stippling examples: (a) Secord’s [137] approach (2000 dots); (b) Kopf et al.’s [78] non-repetitive stippling (7700 dots).

Fig. 10. Video stylization driven by coherent segmentation. (a) 3D approach. (b) 2D+t approach. (c) Cartoon and painterly styles arising from rotoscoping the coherent regions [157]. (d) Example of coherent painterly rendering [157].

Full text

HAL Id: hal-00781502
https://hal.inria.fr/hal-00781502
Submitted on 19 Jul 2013
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
State of the ”Art”: A Taxonomy of Artistic Stylization
Techniques for Images and Video
Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, Tobias Isenberg
To cite this version:
Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, Tobias Isenberg. State of the ”Art”: A Tax-
onomy of Artistic Stylization Techniques for Images and Video. IEEE Transactions on Visualization
and Computer Graphics, Institute of Electrical and Electronics Engineers, 2013, 19 (5), pp.866-885.
�10.1109/TVCG.2012.160�. �hal-00781502�

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 19, NO. 5, MAY 2013 (AUTHORS’ VERSION) 1
State of the ‘Art’: A Taxonomy of Artistic
Stylization Techniques for Images and Video
?
Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg
Abstract
—This paper surveys the field of non-photorealistic rendering (NPR), focusing on techniques for transforming 2D input
(images and video) into artistically stylized renderings. We first present a taxonomy of the 2D NPR algorithms developed over
the past two decades, structured according to the design characteristics and behavior of each technique. We then describe
a chronology of development from the semi-automatic paint systems of the early nineties, through to the automated painterly
rendering systems of the late nineties driven by image gradient analysis. Two complementary trends in the NPR literature are then
addressed, with reference to our taxonomy. First, the fusion of higher level computer vision and NPR, illustrating the trends toward
scene analysis to drive artistic abstraction and diversity of style. Second, the evolution of local processing approaches toward
edge-aware filtering for real-time stylization of images and video. The survey then concludes with a discussion of open challenges
for 2D NPR identified in recent NPR symposia, including topics such as user and aesthetic evaluation.
Index Terms—Image and Video Stylization, Non-photorealistic Rendering (NPR), Artistic Rendering.
✦
1 INTRODUCTION
A
S the advent of photography stimulated artistic
diversity in the late 19
th
century, so did the suc-
cesses of photorealistic computer graphics in the early
nineties motivate alternative techniques for rendering in
non-photorealistic styles. Two decades later, the field of
non-photorealistic rendering (NPR) has expanded into a
vibrant area of research covering a plethora of expressive
rendering styles for the visual communication: exploded
diagrams [88], false color [124], [126], and artistic styles
such as painterly [10], [168] and constrained palette
rendering [106], [167]. It is this latter category of artistic
rendering (AR) that forms the subject of this survey;
specifically, techniques focusing on artistic stylization
of two-dimensional content (photographs and video) to
which we refer as image-based artistic rendering (IB-AR).
IB-AR’s origins reach back to seminal works exploring
the emulation of traditional artistic media and styles [25],
[47], [55], [90], [130]. Today, IB-AR has diversified into a
highly cross-disciplinary activity, which builds upon com-
puter vision (CV), perceptual modeling, human computer
interaction (HCI), and computer graphics. Many classic
IB-AR problems have been found to closely relate to long-
standing problems in computer graphics or computer
vision; for example, video cartooning [21], [156] and its
relationship to video matting and automated rotoscoping
[2]. In many cases computer graphics problems have
benefited from or motivated entirely new computer vision
•
J. E. Kyprianidis is with the Computer Graphics Systems Group of the
Hasso-Plattner-Institut, University of Potsdam, Germany.
•
T. Wang and J. Collomosse are with the Centre for Vision, Speech and
Signal Processing, University of Surrey, UK.
•
T. Isenberg is with the University of Groningen’s Johan Bernoulli
Institute, the Netherlands, and with DIGITEO/CNRS/INRIA, France.
?
This is the authors’ version of the work. The definitive version was
published in IEEE Transactions on Visualization and Computer Graphics.
Vol. 19, No. 5, pp. 866–885, 2013. doi: 10.1109/TVCG.2012.160.
research. Similarly, the goal of much IB-AR research—
that of producing a creative or artistic tool—demands a
careful, user-led HCI design process.
Despite several years of discipline convergence and the
resulting improvements in aesthetic quality and diversity,
there have been few surveys of the IB-AR literature
in the past decade. Common references for IB-AR are
the texts of Gooch and Gooch [42] and Strothotte and
Schlechtweg [148], both of which surveyed pre-2000
techniques (Sec. 3). The majority of other survey material
takes the form of conference tutorials; yet these primarily
focus upon illustrative visualization [95] or NPR for 3D
graphics and games [100]. This survey follows up a recent
tutorial [18] by some of the authors at Eurographics 2011,
prior to which the most recent major conference tutorials
on the topic were by Hertzmann et al. [95] in 2003 and
Green et al. [44] in 1999. Also, a number of web-based
curated bibliographies are available via Reynolds [127]
(to 2004), Schlechtweg [133] (to 2007), and Stavrakis [145].
This article delivers a comprehensive view of the
IB-AR landscape, covering classical and contemporary
techniques while offering two perspectives. First, we
provide an up-to-date taxonomy of IB-AR techniques in
which algorithms are grouped according to the family of
techniques used (e. g., nonlinear filters, region segmenta-
tion) or design characteristics (e. g., local greedy, or global
optimization approaches to rendering).
Second, we present IB-AR’s development in chrono-
logical order, from the early nineties to the modern day
(c. 2011), to reflect the contemporaneous development
of techniques clustered together in our taxonomy; for
example local methods, followed later by global methods.
We first document ‘classical’ (pre-2000) IB-AR and so
introduce the key concepts and algorithms that continue
to underpin and influence more contemporary methods
(Sec. 3). These classical algorithms focused on the stroke-
based rendering (SBR) paradigm [47], [58] with increasing

2 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 19, NO. 5, MAY 2013 (AUTHORS’ VERSION)
1980 1990 1997 1998 2002 2004 2005 2006 2008 2010
NPAR
2010
Grand
Challen-
ges
Video
painting
Litwinowicz’97
Semi-automatic
painting systems
Haeberli’90
Fully automatic
painting
Hertzmann’98
Treveatt’97
Perceptual UI and
segmentation
DeCarlo’02
Automatic
perceptual
Collomosse’05
Space-time
video
Wang’04
Collomosse’05
GPU-based image
processing
Winnemöller’06
Kang’07/’09
Kyprianidis’08/’09
Late 1980s
Advances
in media
emulation
Strassmann’86
User evaluation
Isenberg’06
Fig. 1. Chronology of IB-AR development. From the semi-automated SBR systems of the early nineties, to increasingly automated systems
drawing upon image processing. Later the aesthetic gamut is enhanced through more sophisticated computer vision and edge-aware filtering.
Recently attention returns to user interaction, raising new questions around the evaluation of aesthetics and usability.
levels of automation and sophistication in stroke place-
ment and driven by low-level image processing (typically
the Sobel operator).
Next, we describe how the early convergence of
computer graphics and image processing developed,
enabling IB-AR to draw increasingly upon the more
sophisticated image analysis offered by contemporary
computer vision algorithms (Sec. 4). One consequence of
the increasingly sophisticated interpretation or ‘parsing’
of the image was a divergence from SBR to alternative
forms of rendering primitives: the use of regions and
tiles which, in turn, unlocked greater diversity in the
gamut of styles available to IB-AR. In line with the
trend toward more complex image analysis, we also
observe IB-AR to be defined increasingly as a goal-
directed task—drawing upon global optimization rather
than local approaches. Although these goals were initially
defined at the low level of image artifacts (e. g., image
gradient), the description of these goals later evolved to
include higher level concepts such as perceptual salience
measures and even emotional or ‘affective’ contexts.
In parallel with the trend toward more sophisticated
scene analysis, IB-AR benefited from the emerging
popularity of anisotropic and edge-preserving forms of
filters in computer graphics (Sec. 5). On the one hand,
such operations lacked high-level image ‘understanding’,
limiting their artistic gamut to painterly, sketchy, and
cartoon styles. On the other hand, their simplicity led
to real-time speeds on GPU hardware, making them
practical for video processing—and applicable to footage
(e. g., water, smoke, fur) that is otherwise challenging to
parse using vision methods such as segmentation.
Concluding, we catalog a number of challenges that
remain outstanding in IB-AR (Sec. 6).
2 TAXONOMY OF IB-AR TECHNIQUES
Early prototype IB-AR systems followed the SBR
paradigm and synthesized artistic renderings by incre-
mentally compositing virtual brush strokes whose color,
orientation, scale, and ordering were derived from semi-
[47] or fully automated processes [55], [90], [151]. The
aesthetics of the output generated by a SBR algorithm
is, therefore, a function of both the media simulation
applied to render each brush stroke and the process by
which strokes are positioned and their attributes are set
(referred to hereafter as the stroke placement algorithm).
Although sometimes described simultaneously in early IB-
AR papers, the problems of media emulation and stroke
placement may be considered de-coupled. The curved
spline strokes placed by Hertzmann’s [55] algorithm
could be rendered by sweeping various brush models
along their trajectories, to emulate thick oil paint, crayon,
charcoal, or pastel, to name but a few different media.
It is, therefore, not surprising that IB-AR has evolved in
parallel with increasingly sophisticated media emulation
models; from simple simulations of hairy brushes [146]
to full multi-layered models of pigment diffusion and
bi-directional transfer between brush and canvas [25].
A detailed exposition of media simulation warrants a
survey in its own right, but in this work we focus only
on the problem of stroke placement, or more generally,
the placement of artistic rendering primitives (regions,
strokes, stipples, tiles). We also survey nonlinear filters
that introduce an anisotropy that conveys the impression
of stroke placement. Accordingly, our taxonomy avoids
the categorization of IB-AR purely in terms of media
(painterly, sketch, cartoon shading) and instead clusters
the space of IB-AR algorithms by the elementary render-
ing primitive or stylization mechanism employed. We
then expand the lower branches of the taxonomy by
considering similarities in the nature of the algorithm;
local approaches vs. global arrangement strategies, or
approaches that address the rendering of outlines vs. the
interior of image regions.
2.1 Stroke-based Rendering (SBR)
SBR algorithms cover a 2D canvas with atomic rendering
primitives according to some process or desired end goal,
designed to simulate a particular style. In many SBR
algorithms these primitives are the eponymous virtual
brush stroke, but the definition of SBR has diversified to
primitives including tiles, stipples and hatch marks [58].
2.1.1 Brush Stroke Techniques
The most prevalent form of IB-AR are perhaps SBR
algorithms using either short dabs of paint, or long
curved brush strokes as rendering primitives. The process
of covering the canvas can be categorized broadly as
local or global. Local approaches typically drive stroke
placement decisions based on the pixels in the spatial
neighborhood of the stroke; this can be explicit in the
algorithm (e. g., image moments within a window [140],
[151]) or implicit due to a prior convolution (e. g., Sobel
edges). An alteration to the image would thus affect
only strokes in the locality. Global methods optimize

KYPRIANIDIS et al.: A TAXONOMY OF ARTISTIC STYLIZATION TECHNIQUES FOR IMAGES AND VIDEO 3
Stroke-based Rendering for Image Approximation
Brush Stroke Techniques
Local
User Interaction
Low
Level
Haeberli’90 [47]
Salisbury’96 [128]
Salisbury’97 [130]
Curtis’97 [25]
Gooch’04 [43]
Grubert’08 [45]
Lin’10 [89]
Kagaya’11 [66]
O’Donovan’11 [108]
Perceptual
Measure
Santella’02 [131]
Automatic
Low Level
Image
Haggerty’91 [48]
Treavett’97 [151]
Salisbury’97 [130]
Hertzmann’98 [55]
Shiraishi’00 [140]
Sziranyi’00 [150]
Wen’06 [161]
Video
Litwinowicz’97 [90]
Hertzmann’00 [60]
Kovacs’02 [79]
Hays’04 [52]
Park’07 [122]
Lu’10 [94]
Perceptual
Measure
Collomosse’02 [15]
Collomosse’05 [17]
Shugrina’06 [141]
Colton’08 [22]
Global
User-guided
Emphasis
Hertzmann’01 [56]
Tresset’05 [152]
Automatic
Emphasis
Szirányi’01 [150]
Collomosse’05 [21]
Mosaicking & Tiling
Still
Hausner’01 [51]
Kim’02 [73]
Dobashi’02 [32]
Elber’03 [37]
Di Blasi’05 [31]
Faustino’05 [39]
Schlechtweg’05 [134]
Orchard’08 [110]
Xu’07 [166]
Xu’08 [167]
Hurtut’09 [63]
Animated
Klein’02 [76]
Smith’05 [142]
Dalal’06 [27]
Kang’11 [67]
Tonal Depiction
Stippling
Local
Single
Resolution
Ulichney’87 [153]
Ostromoukhov’93 [113]
Ostromoukhov’94 [117]
Ostromoukhov’99 [114]
Ostromoukhov’99 [116]
Multiple
Resolution
Streit’98 [147]
Global
Spatial
Constraint
Deussen’00 [30]
Secord’02 [137]
Hiller’03 [61]
Schlechtweg’05 [134]
Kopf’06 [78]
Mould’07 [105]
Vanderhaeghe’07 [154]
Structure and
Spatial Constraint
Kim’08 [72]
Pang’08 [118]
Kim’09 [74]
Martin’11 [98]
Li’11 [87]
Hatching and
Line Art
Salisbury’94 [129]
Dafner’00 [26]
Pedersen’06 [123]
Pang’08 [118]
Mi’09 [103]
Inglis’11 [64]
Region-based Techniques
Image
Fill
Gooch’02 [41]
Mould’03 [104]
O’Donovan’06 [109]
Setlur’06 [139]
Shugrina’06 [141]
Xu’08 [167]
Form/Shape
Salisbury’96 [128]
Salisbury’97 [130]
Gooch’04 [43]
Grubert’08 [45]
Song’08 [144]
Composition
Collomosse’03 [16]
Hall’07 [49]
Hierarchical
DeCarlo’02 [29]
Bangham’03 [5]
Mould’08 [106]
Zeng’09 [168]
Zhao’10 [169]
Video
Appearance
2D+t
Agarwala’02 [1]
Collomosse’03 [20]
Agarwala’04 [2]
Collomosse’05 [21]
Bousseau’06 [9]
Bousseau’07 [10]
Wang’10 [157]
Kagaya’11 [66]
O’Donovan’11 [108]
3D
Wang’04 [156]
Lin’10 [89]
Motion
Stylization
Collomosse’03 [19]
Smith’05 [142]
Liu’05 [91]
Wang’06 [155]
Example-based Techniques
Color
Reinhard’01 [126]
Neumann’05 [107]
Xiao’09 [165]
Pouli’11 [124]
Texture
Hertzmann’01 [59]
Ashikhmin’03 [4]
Hashimoto’03 [50]
Kim’09 [74]
Lee’10 [86]
Martin’11 [98]
Zhao’11 [170]
Image Processing and Filtering
Spatial Domain
Outlines
First
Derivative
Orzan’07 [111]
Second
Derivative
Gooch’04 [43]
Winnemöller’06 [164]
Kang’07 [69]
Kyprianidis’08 [82]
Kang’09 [70]
Winnemöller’11 [163]
Content
Anisotropic
Diffusion
Winnemöller’06 [164]
Kang’07 [69]
Kang’08 [68]
Kyprianidis’08 [82]
Kang’09 [70]
Kyprianidis’11 [83]
Local
Statistics
Papari’07 [119]
Kyprianidis’09 [84]
Kyprianidis’11 [81]
Morphological
Filtering
Bousseau’06 [9]
Bousseau’07 [10]
Papari’09 [120]
Criminisi’10 [24]
Gradient Domain
Orzan’07 [111]
Bhat’10 [8]
Fig. 2. Taxonomy of IB-AR techniques.
the placement of all strokes to minimize some objective
function. Various strategies have been applied from
snake relaxation [56], to evolutionary algorithms [17],
and Monte-Carlo optimization [150]. In all cases the
desired objective relates to retention of detail, for example,
encouraging maximal retention of visual detail [56], [150]
using low-level operators (e. g., Sobel gradient) or higher-
level measures such as image salience to retain only
perceptually important detail [17].
On the more heavily populated ‘local’ branch of the
SBR taxonomy, we partition algorithms into user-assisted
and automatic processes—the former typically pre-dating
the latter, pointing to a trend toward automation post-
nineties. The mechanism behind the automation can, as
with ‘global’ SBR, be divided into lower- and higher-level
analysis according to the definition of the ‘importance’
field that guides the emphasis of features in the artwork.
In the parallel SBR branch of semi-automated (i. e., user-
assisted) algorithms, the low/high-level distinction is
again mirrored; with early techniques relying on image
filters to orient brush strokes [47] and later work—pre-
dating automated measures for emphasis—using gaze
trackers to directly harness the perceptual measures
inherent in the human visual system [131]. In some recent
automated algorithms, stroke placement is influenced by
even higher-level contextual parameters such as emotion
and mood [22], [141]. Most recently, there has been a trend
back toward interaction, producing semi-automated tools
for painterly video that enable keyframing of the fields
used to arrange strokes [66], [89], [108].
For automatic techniques, a clear distinction can be
made between those operating over images versus video
content. Video extensions of SBR are non-trivial as
strokes must not scintillate (flicker) and their motion
must match the underlying video content. In the SBR
branch of the taxonomy this problem has largely been
addressed—though by no means solved—using optical
flow. Elsewhere, nonlinear filters and segmentation have
been applied.
2.1.2 Mosaicking, Tiling and Stippling
A further sub-category of SBR aims to approximate the
image using a medium other than colored pixels or
paint, packing image regions with a multitude of atomic
rendering primitives. The techniques approximate the
image content by either (i) stippling, the distribution of
small points (stipples) often for the purpose of tonal
depiction; (ii) hatching, the use of line patterns or curves
for the same; and (iii) mosaicking algorithms that pack
small tiles together.
Stippling IB-AR techniques are closely related to digital
half-toning and dithering algorithms that locally approxi-
mate regions using dot patterns, either with the sole goal
of representing a local brightness or with an additional
artistic intent [114]. Many early half-toning techniques
developed heuristically informed greedy strategies for
populating regions with stipples to avoid artifacts due
to aliasing. Such techniques operate at either single
or multiple scales, placing dots using local decision
making. This culminated most recently in techniques
designed to emphasize image structure [118], following

4 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 19, NO. 5, MAY 2013 (AUTHORS’ VERSION)
the trend toward perceptual analysis in SBR. In contrast
to half-toning, stippling does not simply decide whether
to use a black or a white pixel on a regular grid
but tries to place larger dots, with the shared goal to
represent the brightness and to (typically) avoid visible
patterns. Early stippling used a number of brush-based
techniques [30]. However, much as local SBR painterly
approaches evolved into global relaxation approaches, so
image stippling began to adopt a more global strategy
for stipple placement. Recently, goals in stippling are to
capture and replicate aspects of the stippling style of
artists [74], [98] or to be able to reproduce non-repetitive
patterns [78]. A smaller subset of IB-AR explored the
approximation of images using lines and curves. Aside
from dedicated image-based hatching approaches [129],
some techniques grow labyrinthian patterns using space-
filling curves [26] or reaction diffusion processes [123]
that adapt to the intensity of the image.
Artistic mosaicking algorithms are closely related to
packing problems, and so are approached almost uni-
versally as global optimization problems. While packing
strategies vary widely, they can be categorized into those
obeying purely spatial or spatio-temporal constraints.
The latter are especially challenging since a balance
must be maintained between a faithful approximation of
frame content and the introduction of flicker (temporal
incoherence) due to frequent update of the tile or glyph
chosen to represent a particular spatial region.
2.2 Region-based Techniques
Much as SBR in the 1990s relied increasingly on low-
level image processing (e. g., intensity gradient, moments,
optical flow), a trend post-2000 was the emergence of
mid-level computer vision in IB-AR. Segmentation is
frequently incorporated as step toward parsing image
structure, enabling the adaptation of rendering according
to the content in regions. In some techniques, SBR
algorithms are applied to render the interiors of regions
independently [41], [141], [157]. However, the use of
regions as rendering primitives in their own right has
also given rise to additional styles including cartoon
‘flat’ shading [21], [156], new materials such as stained
glass [104], [139], felt [109], and even emulation of abstract
artistic styles [16].
For images, we categorize region-based approaches
into those considering the arrangement of rendering
primitives (e. g., strokes) within the interiors of regions
and those manipulating shape, form, and composition of
regions. A further category explores techniques based on
image pyramids. Various interactive techniques (human
gaze-trackers [29], importance maps [5]) are used to
browse a region containment hierarchy constructed by
segmenting successively lower resolution versions of the
source image. An image can be rendered at a high level of
abstraction by drawing only coarse large regions near the
top of the hierarchy, or particular regions can be rendered
in greater detail at lower levels. This enables local control
over the level of detail. Such methods were among the
first region-based IB-AR algorithms and are significant by
being among the first to consider perceptual importance.
The consideration of regions in IB-AR has also ben-
efited video stylization, offering an alternative to SBR
techniques dependent on optical flow. Video segmenta-
tion is a well-studied problem in computer vision and
is broadly separated into two categories: techniques that
segment frames independently and associate regions over
time (2D
+t
) and those segmenting video as a spatio-
temporal
(x
,
y
,
t)
volume (3D). Both methodologies have
seen applications to IB-AR for the purpose of cartooning
or otherwise stylizing the appearance of video. All
techniques share the observation that once video has
been coherently segmented into regions (a non-trivial
problem), the problem of hatching, sketching, or painting
with temporal coherence can be solved by attaching
strokes to a rigid [21] or deforming [2] region. This frames
the problem of IB-AR as one of automated rotoscoping.
Finally, when considering regions, it is possible to track
and analyze the motion of objects. This gives rise to
a complementary form of video stylization—that of
artistically manipulating object motion.
2.3 Example-based Rendering
Most IB-AR algorithms encode a set of heuristics, typ-
ically emulating artistic practice with the goal of faith-
fully depicting a prescribed style. A complementary
approach to IB-AR—example-based rendering pioneered
by Hertzmann et al. [59]—learns the mapping between an
exemplar pair: a source image and an artist’s rendering
of that image. The learned mapping can then be applied
to render arbitrary images in the exemplar style.
Example-based rendering (EBR) can be categorized
as performing either texture or color transfer. Color
EBR typically performs a piecewise mapping between
the color histograms of two images to effect a non-
photorealistic recoloring. Often there is only weak enforce-
ment of spatial coherence in the color mapping process.
By contrast, texture-based EBR shares similarities with
patch-based texture in-filling techniques [35], [36], which
seek to fill holes in images by searching for visually
similar patches elsewhere in the image. However, in
the case of EBR the patches are not matched within
the source image to be rendered but instead within the
exemplar source image. The corresponding patch from
the exemplar artistic image is then pasted into place in
the output rendering. As with texture in-filling, a careful
balance must be maintained between fidelity of the patch
matching and the spatial coherence in the rendering.
2.4 Image Processing and Filtering
Many image processing filters have been explored for
IB-AR but few have been recognized so far to produce
interesting results from an artistic point of view. This is
probably because these filters are often concerned with
the restoration and recovery of photorealistic imagery. By
contrast, IB-AR generally aims for simplification.

Frequently asked questions

Q1. What are the contributions in "State of the ”art”: a taxonomy of artistic stylization techniques for images and video" ?

This paper surveys the field of non-photorealistic rendering ( NPR ), focusing on techniques for transforming 2D input ( images and video ) into artistically stylized renderings. The authors first present a taxonomy of the 2D NPR algorithms developed over the past two decades, structured according to the design characteristics and behavior of each technique. The authors then describe a chronology of development from the semi-automatic paint systems of the early nineties, through to the automated painterly rendering systems of the late nineties driven by image gradient analysis. 

Q2. What have the authors stated for future works in "State of the ”art”: a taxonomy of artistic stylization techniques for images and video" ?

They express the view that ( 6 ) remains the most promising direction ; that NPR should “ not just imitate and emulate styles of the past but create styles for the future. ” They also observe that Salesin ’ s research questions regarding definitions of aesthetics and the artistic Turing test should be given equal weight in terms of new artistic styles emerging as a consequence of NPR. Further positions regarding directions for NPR were presented at NPAR 2010 by DeCarlo and Stone [ 28 ] and Hertzmann [ 54 ]. 

Q3. What are the main approaches to such example-based rendering?

There are two main approaches to such example-based rendering (EBR): methods seeking to perform texture transfer (typically performed by modulating the luminance channel) and those focusing on color transfer leaving texture constant. 

Q4. What is the effect of the bilateral filter on low-contrast images?

The bilateral filter smoothes low-contrast regions while preserving high-contrast edges, but may fail for highcontrast images where either no abstraction is performed or salient visual features may be removed. 

Q5. What is the meaning of the term "Extensions of photomosaicking"?

Work approximating images with irregular tiles (e. g., jigsaw image mosaics [73]) can be considered extensions of photomosaicking. 

Q6. What is the common method of applying morphological smoothing to watercolor paintings?

Since watercolor paintings typically have light colors, Bousseau et al. [10] proposed to swap the order of the morphological operators and apply closing followed by opening. 

Q7. How many man-hours of manual correction to optical flow fields were required to produce the short?

Green et al. [44] report that over 1000 man-hours of manual correction to optical flow fields were required to produce the short painterly scenes in the movie. 

Q8. What is the way to preserve the visual richness of color photographs?

Qu et al. [125], for example, preserve the visual richness of color photographs by applying a range of stippling and related bitonal techniques to different regions in the image. 

Q9. What is the common term used for morphological smoothing?

These are related to order-statistics filters and applying opening and closing in sequence results in a smoothing operation that is often referred to as morphological smoothing. 

Q10. What is the main idea behind the IB-AR algorithm?

Initially proposed by DeCarlo and Santella [29] as a mechanism for interactive abstraction of photographs (Sec. 4.1), image segmentation has become a cornerstone of many automatic IB-AR algorithms that make rendering decisions based on mid-level structure parsed from the image. 

Q11. What are the main approaches to the transfer of artistic texture?

The majority of artistic EBR algorithms focus on the transfer of artistic texture, and borrow from the nonparametric patch-based methods used for texture synthesis and photo in-painting. 

Q12. What are the different types of techniques used to browse a region containment hierarchy?

Various interactive techniques (human gaze-trackers [29], importance maps [5]) are used to browse a region containment hierarchy constructed by segmenting successively lower resolution versions of the source image. 

Q13. What is the motivation of contemporary IB-AR work?

Although a few IB-AR systems of the early nineties cited their motivation as emulating the artist (i. e., passing the artistic Turing test), the frequently stated motivation of contemporary IB-AR work is to retain human creativity and to deliver useful tools and new artistic media. 

Q14. What was not present in the final smoothing pass?

Also not present were the iterative application of the DoG filter [69] and the final smoothing pass to further reduce aliasing of edges. 

Q15. What is the technique used to create a sequence of spline control points?

Given a starting or seed pixel, a sequence of spline control points is generated by iteratively hopping between pixels normal to the direction of the image gradient (Fig. 4). 

Q16. What is the definition of a high quality painting?

A high quality painting is deemed to be one that matches the source image as closely as possible, using a minimal number of strokes but covering the maximum area of canvas in paint.