We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

/pdf/image-to-image-translation-with-conditional-adversarial-5058t24et2.pdf

Image-to-Image Translation with Conditional Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

We present a novel approach for image completion that results in images that are both locally and globally consistent. With a fully-convolutional neural network, we can complete images of arbitrary resolutions by filling-in missing regions of any shape. To train this image completion network to be consistent, we use global and local context discriminators that are trained to distinguish real images from completed ones. The global discriminator looks at the entire image to assess if it is coherent as a whole, while the local discriminator looks only at a small area centered at the completed region to ensure the local consistency of the generated patches. The image completion network is then trained to fool the both context discriminator networks, which requires it to generate images that are indistinguishable from real ones with regard to overall consistency as well as in details. We show that our approach can be used to complete a wide variety of scenes. Furthermore, in contrast with the patch-based approaches such as PatchMatch, our approach can generate fragments that do not appear elsewhere in the image, which allows us to naturally complete the images of objects with familiar and highly specific structures, such as faces.

Globally and locally consistent image completion

This work presents a Multi-View Stereo system for robust and efficient dense modeling from unstructured image collections. Our core contributions are the joint estimation of depth and normal information, pixelwise view selection using photometric and geometric priors, and a multi-view geometric consistency term for the simultaneous refinement and image-based depth and normal fusion. Experiments on benchmarks and large-scale Internet photo collections demonstrate state-of-the-art performance in terms of accuracy, completeness, and efficiency.

/pdf/pixelwise-view-selection-for-unstructured-multi-view-stereo-ue2no30piv.pdf

Pixelwise View Selection for Unstructured Multi-View Stereo

We propose a method for semantic parsing of images with regular structure The structured objects are modeled in a densely connected CRF The paper describes how to embody specific spatial relations in a representation called Spatial Pattern Templates (SPT), which allows us to capture regularity constraints of alignment and equal spacing in pairwise and ternary potentials

/pdf/spatial-pattern-templates-for-recognition-of-objects-with-5eszulrb4j.pdf

Spatial Pattern Templates for Recognition of Objects with Regular Structure

We propose a novel method for recognition of structured images and demonstrate it on detection of windows in facade images. Given an ability to obtain local low-level data evidence on primitive elements of a structure (like window in a facade image), we determine their most probable number, attribute values (location, size) and neighborhood relation. The embedded structure is weakly modeled by pair-wise attribute constraints, which allow structure and attribute constraints to mutually support each other. We use a very general framework of reversible jump MCMC, which allows simple implementation of a specific structure model and plug-in of almost arbitrary element classifiers. The MC controls the classifier by prescribing it "where to look", without wasting too much time on unpromising locations.

We have chosen the domain of window recognition in facade images to demonstrate that the result is an efficient algorithm achieving performance of other strongly informed methods for regular structures like grids, while our general model covers loosely regular configurations as well.

/pdf/a-weak-structure-model-for-regular-pattern-recognition-oyiai52u83.pdf

A weak structure model for regular pattern recognition applied to facade images

In this paper we propose a pipeline for accurate 3D reconstruction from multiple images that deals with some of the possible sources of inaccuracy present in the input data. Namely, we address the problem of inaccurate camera calibration by including a method adjusting the camera parameters in a global structure-and-motion problem, which is solved with a depth map for representation that is suitable to large scenes.
Secondly, we take the triangular mesh and calibration improved by the global method in the first phase to refine the surface both geometrically and radiometrically. Here we propose surface energy which combines photoconsistency with contour matching and minimize it with a gradient descent method. Our main contribution lies in effective computation of the gradient that naturally regularization and data terms by employing scale space approach. The results are demonstrated on standard high-resolution datasets and a complex outdoor scene.

/pdf/refinement-of-surface-mesh-for-accurate-multi-view-1xdhp81ypx.pdf

Refinement of Surface Mesh for Accurate Multi-View Reconstruction

Robots are increasingly present in modern industry and also in everyday life. Their applications range from health-related situations, for assistance to elderly people or in surgical operations, to automatic and driver-less vehicles (on wheels or flying) or for driving assistance. Recently, an interest towards robotics applied in agriculture and gardening has arisen, with applications to automatic seeding and cropping or to plant disease control, etc. Autonomous lawn mowers are succesful market applications of gardening robotics. In this paper, we present a novel robot that is developed within the TrimBot2020 project, funded by the EU H2020 program. The project aims at prototyping the first outdoor robot for automatic bush trimming and rose pruning.

TrimBot2020: an outdoor robot for automatic gardening

Place recognition can be achieved by identifying whether a pair of images (a labeled reference image and a query image) depict the same place, regardless of appearance changes due to different viewpoints or lighting conditions. It is an important component of systems for camera localization and for loop closure detection and a widely studied problem for indoor or urban environments. Recently, the use of robots in agriculture and automatic gardening has created new challenges due to the highly repetitive appearance with prevalent green color and repetitive texture of garden-like scenes. The lack of available data recorded in gardens or plant fields makes difficult to improve localization algorithms for such environments. In this paper, we propose a new data set of garden images for testing algorithms for visual place recognition. It contains images with ground truth camera pose recorded in real gardens at different times, with varying light conditions. We also provide ground truth for all possible pairs of images, indicating whether they depict the same place or not. We also performed a thorough benchmark of several holistic (whole-image) descriptors, and provide the results on the proposed data set. We observed that existing descriptors have difficulties with scenes with repetitive textures and large changes of camera viewpoint.

/pdf/tb-places-a-data-set-for-visual-place-recognition-in-garden-1im1l8an6y.pdf

Radim Tylecek

Papers

Spatial Pattern Templates for Recognition of Objects with Regular Structure

A weak structure model for regular pattern recognition applied to facade images

Refinement of Surface Mesh for Accurate Multi-View Reconstruction

TrimBot2020: an outdoor robot for automatic gardening

TB-places: A data set for visual place recognition in garden environments