State of the "Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video
Summary (10 min read)
1 INTRODUCTION
- A S the advent of photography stimulated artistic diversity in the late 19 th century, so did the successes of photorealistic computer graphics in the early nineties motivate alternative techniques for rendering in non-photorealistic styles.
- It is this latter category of artistic rendering (AR) that forms the subject of this survey; specifically, techniques focusing on artistic stylization of two-dimensional content (photographs and video) to which the authors refer as image-based artistic rendering (IB-AR).
- Today, IB-AR has diversified into a highly cross-disciplinary activity, which builds upon computer vision (CV), perceptual modeling, human computer interaction (HCI), and computer graphics.
Late 1980s Advances in media emulation
- From the semi-automated SBR systems of the early nineties, to increasingly automated systems drawing upon image processing.
- Later the aesthetic gamut is enhanced through more sophisticated computer vision and edge-aware filtering.
- Next, the authors describe how the early convergence of computer graphics and image processing developed, enabling IB-AR to draw increasingly upon the more sophisticated image analysis offered by contemporary computer vision algorithms (Sec. 4).
- One consequence of the increasingly sophisticated interpretation or 'parsing' of the image was a divergence from SBR to alternative forms of rendering primitives: the use of regions and tiles which, in turn, unlocked greater diversity in the gamut of styles available to IB-AR.
2 TAXONOMY OF IB-AR TECHNIQUES
- Early prototype IB-AR systems followed the SBR paradigm and synthesized artistic renderings by incrementally compositing virtual brush strokes whose color, orientation, scale, and ordering were derived from semi- [47] or fully automated processes [55] , [90] , [151] .
- The problems of media emulation and stroke placement may be considered de-coupled.
- The curved spline strokes placed by Hertzmann's [55] algorithm could be rendered by sweeping various brush models along their trajectories, to emulate thick oil paint, crayon, charcoal, or pastel, to name but a few different media.
- The authors also survey nonlinear filters that introduce an anisotropy that conveys the impression of stroke placement.
- Accordingly, their taxonomy avoids the categorization of IB-AR purely in terms of media (painterly, sketch, cartoon shading) and instead clusters the space of IB-AR algorithms by the elementary rendering primitive or stylization mechanism employed.
2.1 Stroke-based Rendering (SBR)
- SBR algorithms cover a 2D canvas with atomic rendering primitives according to some process or desired end goal, designed to simulate a particular style.
- In many SBR algorithms these primitives are the eponymous virtual brush stroke, but the definition of SBR has diversified to primitives including tiles, stipples and hatch marks [58] .
2.1.1 Brush Stroke Techniques
- The most prevalent form of IB-AR are perhaps SBR algorithms using either short dabs of paint, or long curved brush strokes as rendering primitives.
- The process of covering the canvas can be categorized broadly as local or global.
- Local approaches typically drive stroke placement decisions based on the pixels in the spatial neighborhood of the stroke; this can be explicit in the algorithm (e. g., image moments within a window [140] , [151] ) or implicit due to a prior convolution (e. g., Sobel edges).
- Various strategies have been applied from snake relaxation [56] , to evolutionary algorithms [17] , and Monte-Carlo optimization [150] .
- In the parallel SBR branch of semi-automated (i. e., userassisted) algorithms, the low/high-level distinction is again mirrored; with early techniques relying on image filters to orient brush strokes [47] and later work-predating automated measures for emphasis-using gaze trackers to directly harness the perceptual measures inherent in the human visual system [131] .
2.1.2 Mosaicking, Tiling and Stippling
- A further sub-category of SBR aims to approximate the image using a medium other than colored pixels or paint, packing image regions with a multitude of atomic rendering primitives.
- The techniques approximate the image content by either (i) stippling, the distribution of small points often for the purpose of tonal depiction; (ii) hatching, the use of line patterns or curves for the same; and (iii) mosaicking algorithms that pack small tiles together.
- Stippling IB-AR techniques are closely related to digital half-toning and dithering algorithms that locally approximate regions using dot patterns, either with the sole goal of representing a local brightness or with an additional artistic intent [114] .
- This culminated most recently in techniques designed to emphasize image structure [118] , following the trend toward perceptual analysis in SBR.
- Aside from dedicated image-based hatching approaches [129] , some techniques grow labyrinthian patterns using spacefilling curves [26] or reaction diffusion processes [123] that adapt to the intensity of the image.
2.2 Region-based Techniques
- Much as SBR in the 1990s relied increasingly on lowlevel image processing (e. g., intensity gradient, moments, optical flow), a trend post-2000 was the emergence of mid-level computer vision in IB-AR.
- For images, the authors categorize region-based approaches into those considering the arrangement of rendering primitives (e. g., strokes) within the interiors of regions and those manipulating shape, form, and composition of regions.
- Both methodologies have seen applications to IB-AR for the purpose of cartooning or otherwise stylizing the appearance of video.
- This frames the problem of IB-AR as one of automated rotoscoping.
- Finally, when considering regions, it is possible to track and analyze the motion of objects.
2.3 Example-based Rendering
- Most IB-AR algorithms encode a set of heuristics, typically emulating artistic practice with the goal of faithfully depicting a prescribed style.
- A complementary approach to IB-AR-example-based rendering pioneered by Hertzmann et al. [59] -learns the mapping between an exemplar pair: a source image and an artist's rendering of that image.
- Color EBR typically performs a piecewise mapping between the color histograms of two images to effect a nonphotorealistic recoloring.
- Often there is only weak enforcement of spatial coherence in the color mapping process.
- The corresponding patch from the exemplar artistic image is then pasted into place in the output rendering.
2.4 Image Processing and Filtering
- Many image processing filters have been explored for IB-AR but few have been recognized so far to produce interesting results from an artistic point of view.
- Among the filtering approaches to IB-AR, the authors distinguish two major categories depending on the domain the techniques operate on.
- Most approaches that have been derived from classical image processing techniques fall into this category.
- The authors adopt the usual distinction between first and second order derivative methods.
- Techniques based on the bilateral filter fall into the anisotropic diffusion category since the bilateral filter can be interpreted as fast filter-based approximation of anisotropic diffusion.
3 CLASSICAL STYLIZATION ALGORITHMS
- IB-AR arguably began to gain momentum in the early 1990s with semi-automated paint systems such as Haeberli's [47] that enabled photos to be transformed into impressionist-like paintings with minimal labor.
- People would click on a photo, each click prompting the generation of a virtual brush stroke of the underlying image color.
- The introduction of noise prior to this process results in stroke color variation reminiscent of an impressionist painting.
- Haeberli's system [47] was motivated by a desire to enrich digital painting by automating the color selection process (reducing the 'time to palette').
- This concept of stroke-based rendering (SBR) [58] underpins almost all of the IB-AR work developed in the nineties, which sought increasingly to automate and to enhance the sophistication of stroke placement.
3.1 Local Algorithms for Stroke Placement
- Haeberli's framework [47] automates the selection of stroke color, and for non-circular brush strokes can also decide stroke orientation by painting strokes orthogonal to the intensity gradient in the source image.
- The system relies upon the user to determine the order and Fig. 3 .
- Adapting Haeberli's framework [47] to randomly assign stroke size and order leads to loss of salient detail [48] and motivated later the use of image processing operators for stroke placement.
- The size and sequencing of stroke overpainting is crucial to producing results with an acceptable aesthetic and without any loss of salient detail.
- With this solution strokes can be painted at sizes disproportionate to the features they represent.
3.1.1 Early Pen-and-Ink Hatching Algorithms
- Early semi-automated systems for rendering in pen-andink and cross-hatched styles follow a similar pattern of development.
- Salisbury et al. [129] developed a semiautomated hatching system that oriented textures according to the underlying image gradients, much as Haeberli's oriented brush strokes [47] .
- A multi-scale extension of the system [128] offered aesthetic improvements in the viewing of hatching patterns at multiple scales.
- Regionbased editing and manipulation of the underlying image gradient was later introduced by Salisbury et al. [130] , enabling discontinuities and swirl effects to be manually introduced, improving tool expressiveness.
3.1.2 Early Painterly Rendering Algorithms
- The first automatic solution to IB-AR described in full detail within the literature was a painterly rendering tool proposed by Litwinowicz [90] .
- Strokes are rectangular, and oriented using Sobel gradients as done previously [47] , [48] .
- The clipping process results in crisp edges that mitigate against strokes from unimportant regions over-painting more important regions.
- Later, Hays and Essa [52] adopted a similar technique for interpolation in their video painting algorithm.
- A notable exception is Treavett and Chen's [151] adoption of image moments computed local to each pixel.
3.2 Local Coarse-to-fine IB-AR Algorithms
- Constant-sized rectangular strokes, even after clipping, generate an artificial regularity that could degrade the resulting aesthetic.
- Each scale of the pyramid corresponds to a layer in the painting; the coarsest scale is painted as the first layer with large strokes.
- The latter enables texturing or bump-mapping of the stroke to produce a convincing oil-paint effect [57] .
- Multi-resolution rendering image analysis was adopted soon afterwards by half-toning algorithms that similarly distributed rendering primitives (stipples or short lines) across a low-pass pyramid [147] .
3.3 Video Stylization
- The extension of SBR to video stylization is non-trivial, since independent per-frame rendering of the image sequence will result in a distracting flickering or scintillation in the animation.
- It is, therefore, desirable that: 1) the motion of brush strokes both matches the motion of the underlying video content, and 2) the animated sequence is flicker free.
- In the case of overly dense regions, strokes are deleted at random until the density reaches acceptable levels.
- When a reliable optical flow estimate is available, the technique performs well.
- In the late nineties real-time optical flow was impractical, leading Hertzmann and Perlin [60] to present an alternative video painting technique amenable to interactive rendering: a "Living Painting.".
4 VISION FOR STYLIZATION
- An increasing reliance upon local image processing techniques (predominantly the Sobel gradient operator) was instrumental in transforming the interactive IB-AR systems of the early nineties into fully automatic rendering systems.
- Continuing this trend towards deeper image analysis, a major trend post-nineties was the tendency to rely increasingly upon higher-level computer vision to guide artistic rendering.
- This trend began with the adoption of mid-level computer vision methods, specifically the use of image segmentation.
4.1 Perceptual Measures for Stylization
- DeCarlo and Santella [29] were among the first to apply image segmentation in IB-AR.
- Images were segmented using a variant of mean-shift [23] , [101] at multiple downsampled resolutions.
- This hierarchical representation enabled an image to be rendered in a highly abstract form (using coarse regions from the top of the pyramid), or for certain regions to be locally decomposed into finer grain regions by descending the hierarchy.
- Such techniques scaled strokes in inverse proportion to edge magnitude and so conserved all fine (i. e., high frequency) detail in the painting unless interactively down weighted, e. g., using manually specified masks [56] .
- While Decarlo and Santella harnessed the power of the human visual system to generate their importance maps, a number of IB-AR algorithms were developed using fully automated measures of salience to drive emphasis in renderings.
4.2 Artistic Rendering as a Global Optimization
- Most IB-AR algorithms in the late 1990s treated painting as a local process: pixels in the image are examined in turn and strokes placed according to various heuristics.
- Each stroke is placed according to information in its local spatial neighborhood only.
- By contrast, global approaches to IB-AR iteratively optimize the position of rendering elements (e.g. brush strokes, or stipples) to minimize some objective function defined to describe the 'optimality' according to one or more heuristics.
- It was not until a decade later that the first algorithmic solution was described for painterly rendering [56] .
4.2.1 Global Approaches to SBR: Brush-based
- Hertzmann [56] extended his local curved stroke painterly algorithm [55] by treating each stroke as an active contour or snake.
- A snake is a piecewise curve, whose control points are iteratively updated to minimize an energy function.
- In Hertzmann's optimization [56] , a single painting is created from the source photograph and iteratively updated to converge toward an aesthetic ideal.
- The weights ω 1..4 control the influence of each quality attribute and are determined empirically.
- A similar model of stroke redundancy was presented in the global approach of Szirányi et al. [150] , using a Monte-Carlo Markov Chain (MCMC) optimization.
4.2.2 Global Approaches to SBR: Tonal Depiction
- A purely tonal IB-AR depiction is achieved using stippling.
- In practice, however, random noise can only partially remove artifacts.
- By contrast, most stippling algorithms seek to minimize such artifacts-and in this sense such techniques are related to image-based hatching approaches [115] , [130] that also take structure into account in placing marks.
- Kim et al. [74] generate stipple dot distributions with the same statistical properties as those created by artists.
- The authors now describe this area of example based rendering in greater detail.
4.3.1 Texture by Analogy
- The majority of artistic EBR algorithms focus on the transfer of artistic texture, and borrow from the nonparametric patch-based methods used for texture synthesis and photo in-painting.
- Such methods (e. g., due to Efros et al. [35] , [36] ) in-fill from the edges of 'holes' in an image-iteratively copying patches from elsewhere in the image that share similarity with adjacent texture.
- PCA is used to reduce the dimensionality of the search, which can be time-consuming for ANN over large dimensions (patch sizes).
- Only the normalized luminance channel is considered.
- Video EBR is challenging due to the problem of constraining patch choice to satisfy not only local and global spatial coherence terms but also temporal coherence.
4.3.2 Color Transfer
- Manipulating color tone can affect the mood of an artistically rendered image, and forms a useful addition to the IB-AR toolbox.
- Early approaches model the histogram as unimodal, equalizing the mean and variance of the source and target image (either as three 1D per-channel operations [126] or in 3D space [107] ).
- More sophisticated approaches adapt to edges by considering image gradients [165] or perform matching of the histogram at multiple scales [124] .
4.4 Region-based IB-AR Algorithms
- Initially proposed by DeCarlo and Santella [29] as a mechanism for interactive abstraction of photographs (Sec. 4.1), image segmentation has become a cornerstone of many automatic IB-AR algorithms that make rendering decisions based on mid-level structure parsed from the image.
- The ability to harness structural representations of image content led to greater diversity of style (unlocking styles such as stained glass rendering or compositional artwork such as pseudo-Cubism).
- Arguably, aesthetics were also open for improvement as style and emphasis could be controlled at a higher level (e. g., regions) rather than in response to low-level features.
4.4.1 Region Painting and Texturing
- The earliest region-based IB-AR algorithms focused on painterly rendering and were essentially SBR algorithms that used the shape of the region rather than an image gradient field (as common in pre-2000 SBR) to guide the placement of strokes [41] , [77] .
- Shugrina et al. [141] filled region interiors with brush strokes aligned with the principal axis but placed brush strokes on the region boundaries for outlines.
- The systems described so far only make use of the color and gradient information within regions.
- The classification drives the type of stroke placed, based on a pre-digitized database of stroke textures from real brushwork mapped to each texture category.
- Variants of flat shading using only black and white were presented by Xu and Kaplan [167] and sought to depict the underlying image tone whilst discouraging connected regions of similar tone.
4.4.2 Deformation and Composition
- Song et al. [144] classify regions into one of several canonical shapes and replace regions with those shapes to create a simplified shape rendering resembling a paper cut out.
- Region deformation was also employed to warp regions into superquadric shapes reminiscent of Cubist renderings [16] .
- This work also re-arranges the position of regions in order to create abstract compositions; arguably styles such as Cubism could not be generated without region-based analysis.
- Shape simplification was also explored by Mi et al. [103] through decomposition into parts rather than substitution with simpler shapes [144] .
4.5 Region Tiling and Packing Algorithms
- A considerable volume of IB-AR literature addresses the arrangement of a multitude of small tiles (from regular shapes to irregular pictograms) to form artistic representations.
- These mosaicking algorithms are typically phrased as optimization problems seeking to maximize coverage of a 2D region, whilst minimizing tile overlap.
- The tile placement is content-aware, penalizing solutions that misalign tiles to cross edges in the image.
- A spatial coherence term is often introduced to encourage smoothly varying scale and orientation over the tiled region.
4.5.1 Photo and Video Mosaics
- The recti-linear tiling of small image thumbnails to approximate a larger image (so called photomosaics) were among the earliest form of synthetic mosaic, inspired by early physical macro-artwork such as Dali's Abraham Lincoln.
- Thumbnails are often chosen to have a semantic connection to the larger image being created, as in Dali's work.
- The IB-AR literature describes optimized search strategies for expedited rendering of photomosaics [6] as well as alternative optimization strategies such as evolutionary search [14] .
- Klein et al. [76] extended photomosaics to video, updating elements of the mosaic to approximate video content whilst penalizing frequent changes of a given tile to prevent flicker.
- Work approximating images with irregular tiles (e. g., jigsaw image mosaics [73] ) can be considered extensions of photomosaicking.
4.5.2 Voronoi Methods
- The earliest mosaic-like renderings relied on Voronoi diagrams constructed from points randomly seeded over the image [47] .
- Dobashi et al. [32] modified this approach to iteratively relax the position of the Voronoi seeds to better approximate the image using a mean-squared error (MSE) between the source and rendered image.
- Faustino et al. [39] place regular tiles instead of relying on Voronoi segments but guide tile placement using Voronoi regions.
- The tiles are scaled in proportion to image size to preserve detail.
- Grundland et al. [46] form Voronoi segments according to both edge strength and image intensity.
4.5.3 Packing and Tessellation methods
- Hausner et al. [51] were the first to address irregular tile shapes through an energy minimization scheme for shape packing.
- Kim et al.'s [73] jigsaw image mosaics (JIM) extended this approach using an active contour based optimization scheme to minimize the energy function to allow moderate tile deformation.
- Branch and bound heuristics are used to improve search efficiency (Fig. 9(d) ).
- The work follows up on an earlier specific case of irregular tiling: calligraphic (text) packing [166] .
- Hurtut et al. [63] combined the principles of texture modeling and mosaicking to learn statistical distributions of tiles.
4.6 Computer Vision for Video Stylization
- A major goal in video stylization is temporal coherence; requiring video to exhibit minimal flicker and the rendering primitives (e. g., strokes) to move with the underlying video content.
- Early algorithms for 2D video stylization were based on per-pixel analysis using optical flow and frame differencing (Sec. 3.3).
- Temporal incoherence is common in such algorithms [60] , [90] since stroke placement decisions are being made on a spatially (per-pixel) and temporally (per-frame) local basis.
- Higher-level analysis of visual structure, e. g., through computer vision can lead to improved coherence.
- The authors now survey two categories of post-nineties algorithms: techniques based on optical flow and segmentation-based methods.
4.6.1 Visual Stylization through Optical Flow
- Approaches that employ optical flow to stylize video were revisited by Hays and Essa [52] .
- To mitigate against temporal incoherence arising from flow estimates, strokes were categorized as weak or strong; the latter in edge areas where gradients are higher.
- Park and Yoon [122] adopted a similar strong-weak categorization.
- Blended texture patches were moved not only forward but also backward in time using a bi-directional estimate of optical flow.
- This mitigated against the cumulative errors inherent in the forward propagation strategies of prior approaches.
4.6.2 Visual Stylization through Segmentation
- Segmentation is now a common component in IB-AR, and by leveraging a similar mid-level representation for video, the consistent motion of strokes within an object can be enforced.
- These benefits come at the cost of generality; not all object are amenable to segmentation (e. g., smoke or water).
- Regions were associated over time using a space-time region adjacency graph that pruned sporadic association to improve stability.
- Painterly and cartoon effects were demonstrated by filling regions with strokes and textures that deform coherently with the boundary.
- In the system of Kagaya et al. [66] , the video is first segmented into spatial-temporal coherent regions.
4.6.3 Motion Stylization
- Video analysis at the region level enables not only consistent rendering within objects, but also facilitates the analysis of object motion.
- Automated methods to generate speed-lines in video require camera motion compensation, as the camera typically pans to track objects.
- This can be approximated by estimating inter-frame homographies.
- Chenney et al. [13] presented early work automatically deforming objects to emphasize motion.
- Other distortions warping the object according to velocity or acceleration emphasized drag or inertia.
5 IMAGE PROCESSING AND FILTERING
- Many of the techniques described in the previous sections are infeasible for real-time rendering and cannot be trivially adapted for multi-core CPUs or GPUs.
- Image processing techniques performing local filtering operations provide an interesting alternative since parallelization and GPU implementations are straightforward in most cases.
- Moreover, a number of filtering techniques have been shown to perform with reasonable temporal coherence when processed frame by frame.
- These advantages, however, come at the expense of style diversity afforded by higher-level interpretation of content.
5.1 Bilateral Filter and Difference of Gaussians
- A fully automatic pipeline for the stylization of cartoon renderings based on images and videos was first proposed in the seminal work by Winnemöller et al. [164] .
- After the conversion to CIELab, the input is iteratively abstracted using the bilateral filter.
- Furthermore, iterative filtering may blur edges resulting in a washedout appearance (Fig. 12(d) ).
- The next section discusses a further popular approach.
5.3 Diffusion and Shock Filter
- Osher and Rudin [112] as well as Weickert [160] recognized the artistic merit of shock filtered imagery, but the work of Kang and Lee [68] was the first to apply diffusion in combination with shock filtering for IB-AR.
- It also creates blurred edges, leading Kang and Lee [68] to perform de-blurring with a shock filter after some MCF iterations, which helps to preserve edges.
- Diffusion that deviates from the local image structure (Fig. 12(f) ). MCF and its constrained variant contract isophote curves to points.
- For this reason, important image features must be protected by a user-defined mask.
- A further limitation is that the technique is not stable against small changes in the input and, therefore, not suitable for perframe video processing.
5.4 Morphological Filtering
- Mathematical morphology (MM) provides a set-theoretic approach to image analysis and processing.
- For grayscale images, dilation is equivalent to a maximum filter and erosion corresponds to a minimum filter.
- Morphological smoothing is applied in Bousseau et al.'s [9] , [10] work on watercolor rendering and in Bangham's et al.'s [5] oil paintings to simplify input images and videos before rendering.
- Because opening and closing are dual, this is equivalent to inverting the output of morphological smoothing applied to the inverted image.
- Then, for every pixel the probability of the pixel's value belonging to a certain cluster is defined.
5.5 Gradient Domain Techniques
- In recent years, gradient domain methods have become very popular in computer vision and computer graphics [3] .
- The basic idea behind such methods is to construct a gradient field representing the result.
- Using scale-space analysis, they extracted a multi-scale Canny edge representation with lifetime and best scale information, which is used to define the gradient field and allows for image operations such as detail removal and shape abstraction.
- Besides being computationally expensive, this technique is also known not to create temporally coherent output for video.
- Bhat et al. [8] have presented a robust optimization framework that allows for the specification of zero-order (pixel value) and first-order (gradient value) constraints over space and time.
6 FUTURE CHALLENGES
- Over the past two decades, IB-AR has delivered many high-quality expressive rendering algorithms and interactive systems.
- As the field gathered momentum, researchers sought to identify the key emerging challenges.
- Artistic rendering or Artistic stylization is also in common parlance, whilst illustrative visualization is being used for approaches in Salesin's third challenge.
- DeCarlo and Stone's discussion focused on visual explanations, that IB-AR can enhance communication by simplification through structural abstraction.
6.1 Evaluation
- Almost one decade since Salesin's panel discussion of this problem, few papers present structured methodologies for evaluation.
- Evaluation work more closely aligned with Salesin's visual communication challenge was proposed by Gooch et al. [43] and Winnemöller et al. [164] in their portrait abstractions.
- Methodologies have been developed to evaluate specific aspects of IB-AR such as visual interest [132] and stippling aesthetics [96] .
- No gold standard methodology has emerged for NPR evaluation.
6.2 Interaction
- Passing the artistic Turing test), the frequently stated motivation of contemporary IB-AR work is to retain human creativity and to deliver useful tools and new artistic media.
- This trend also reflects the limitations of contemporary computer vision and shows that, by carefully designing minimal but well-placed interaction, a high-quality automated visual effects workflow can result.
- Addressing this is especially important if, as Gooch et al. [40] suggest, IB-AR's priority is to develop new artistic media and tools.
- Collaboration with end-users is essential in closing this cycle.
- Connections could be forged with research communities studying computational creativity and evolutionary art.
6.3 Technical directions
- The technical direction of algorithmic research in IB-AR is challenging to predict for a longer term but may develop in the direction of several established mid-term trends.
- Willats and Durand [162] clearly differentiate between such renderings and current IB-AR when writing about the distinction between spatial and depictive systems.
- By contrast, video stylization approaches based on computer vision can perform more aggressive abstraction through mid-level scene parsing (e. g., segmentation) at the cost of generality.
- There is a tendency for complex image processing decisions to become less stable in the presence of noise.
- Overall aesthetics are heavily influenced by media realism, especially in the emulation of traditional artistic styles.
Did you find this useful? Give us your feedback
Citations
4,888 citations
Cites background from "State of the "Art”: A Taxono..."
...For a recent review of the field we refer the reader to [21]....
[...]
2,266 citations
1,286 citations
Cites background from "State of the "Art”: A Taxono..."
...des abundant user controls at runtime, without any modification to the training process. 2. Related Work Style transfer. The problem of style transfer has its origin from non-photo-realistic rendering [28], and is closely related to texture synthesis and transfer [13,12,14]. Some early approaches include histogram matching on linear filter responses [19] and non-parametric sampling [12,15]. These method...
[...]
1,019 citations
972 citations
References
18,095 citations
"State of the "Art”: A Taxono..." refers background in this paper
...the original snakes algorithm [71] moves the curve incrementally closer to an edge over time by minimizing the distance between the curve and edges in the image....
[...]
11,872 citations
11,727 citations
9,602 citations
"State of the "Art”: A Taxono..." refers methods in this paper
...The earliest stipple techniques achieved this goal using Lloyd’s method [93], [99], which computes the Voronoi diagram of a point distribution and...
[...]
...One issue with Lloyd’s method [93], [99] is that it introduces a regularity of stipple dot placement, leading...
[...]
2,972 citations
"State of the "Art”: A Taxono..." refers background in this paper
...By contrast, texture-based EBR shares similarities with patch-based texture in-filling techniques [35], [36], which seek to fill holes in images by searching for visually similar patches elsewhere in the image....
[...]
...[35], [36]) in-fill from the edges of ‘holes’ in an image—iteratively copying patches from elsewhere in the image that share similarity with adjacent texture....
[...]
Related Papers (5)
Frequently Asked Questions (16)
Q2. What have the authors stated for future works in "State of the ”art”: a taxonomy of artistic stylization techniques for images and video" ?
They express the view that ( 6 ) remains the most promising direction ; that NPR should “ not just imitate and emulate styles of the past but create styles for the future. ” They also observe that Salesin ’ s research questions regarding definitions of aesthetics and the artistic Turing test should be given equal weight in terms of new artistic styles emerging as a consequence of NPR. Further positions regarding directions for NPR were presented at NPAR 2010 by DeCarlo and Stone [ 28 ] and Hertzmann [ 54 ].
Q3. What are the main approaches to such example-based rendering?
There are two main approaches to such example-based rendering (EBR): methods seeking to perform texture transfer (typically performed by modulating the luminance channel) and those focusing on color transfer leaving texture constant.
Q4. What is the effect of the bilateral filter on low-contrast images?
The bilateral filter smoothes low-contrast regions while preserving high-contrast edges, but may fail for highcontrast images where either no abstraction is performed or salient visual features may be removed.
Q5. What is the meaning of the term "Extensions of photomosaicking"?
Work approximating images with irregular tiles (e. g., jigsaw image mosaics [73]) can be considered extensions of photomosaicking.
Q6. What is the common method of applying morphological smoothing to watercolor paintings?
Since watercolor paintings typically have light colors, Bousseau et al. [10] proposed to swap the order of the morphological operators and apply closing followed by opening.
Q7. How many man-hours of manual correction to optical flow fields were required to produce the short?
Green et al. [44] report that over 1000 man-hours of manual correction to optical flow fields were required to produce the short painterly scenes in the movie.
Q8. What is the way to preserve the visual richness of color photographs?
Qu et al. [125], for example, preserve the visual richness of color photographs by applying a range of stippling and related bitonal techniques to different regions in the image.
Q9. What is the common term used for morphological smoothing?
These are related to order-statistics filters and applying opening and closing in sequence results in a smoothing operation that is often referred to as morphological smoothing.
Q10. What is the main idea behind the IB-AR algorithm?
Initially proposed by DeCarlo and Santella [29] as a mechanism for interactive abstraction of photographs (Sec. 4.1), image segmentation has become a cornerstone of many automatic IB-AR algorithms that make rendering decisions based on mid-level structure parsed from the image.
Q11. What are the main approaches to the transfer of artistic texture?
The majority of artistic EBR algorithms focus on the transfer of artistic texture, and borrow from the nonparametric patch-based methods used for texture synthesis and photo in-painting.
Q12. What are the different types of techniques used to browse a region containment hierarchy?
Various interactive techniques (human gaze-trackers [29], importance maps [5]) are used to browse a region containment hierarchy constructed by segmenting successively lower resolution versions of the source image.
Q13. What is the motivation of contemporary IB-AR work?
Although a few IB-AR systems of the early nineties cited their motivation as emulating the artist (i. e., passing the artistic Turing test), the frequently stated motivation of contemporary IB-AR work is to retain human creativity and to deliver useful tools and new artistic media.
Q14. What was not present in the final smoothing pass?
Also not present were the iterative application of the DoG filter [69] and the final smoothing pass to further reduce aliasing of edges.
Q15. What is the technique used to create a sequence of spline control points?
Given a starting or seed pixel, a sequence of spline control points is generated by iteratively hopping between pixels normal to the direction of the image gradient (Fig. 4).
Q16. What is the definition of a high quality painting?
A high quality painting is deemed to be one that matches the source image as closely as possible, using a minimal number of strokes but covering the maximum area of canvas in paint.