Inpainting, the technique of modifying an image in an undetectable form, is as ancient as art itself. The goals and applications of inpainting are numerous, from the restoration of damaged paintings and photographs to the removal/replacement of selected objects. In this paper, we introduce a novel algorithm for digital inpainting of still images that attempts to replicate the basic techniques used by professional restorators. After the user selects the regions to be restored, the algorithm automatically fills-in these regions with information surrounding them. The fill-in is done in such a way that isophote lines arriving at the regions' boundaries are completed inside. In contrast with previous approaches, the technique here introduced does not require the user to specify where the novel information comes from. This is automatically done (and in a fast way), thereby allowing to simultaneously fill-in numerous regions containing completely different structures and surrounding backgrounds. In addition, no limitations are imposed on the topology of the region to be inpainted. Applications of this technique include the restoration of old photographs and damaged film; removal of superimposed text like dates, subtitles, or publicity; and the removal of entire objects from the image like microphones or wires in special effects.

Image inpainting

A new algorithm is proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way. In the past, this problem has been addressed by two classes of algorithms: 1) "texture synthesis" algorithms for generating large image regions from sample textures and 2) "inpainting" techniques for filling in small image gaps. The former has been demonstrated for "textures"-repeating two-dimensional patterns with some stochasticity; the latter focus on linear "structures" which can be thought of as one-dimensional patterns, such as lines and object contours. This paper presents a novel and efficient algorithm that combines the advantages of these two approaches. We first note that exemplar-based texture synthesis contains the essential process required to replicate both texture and structure; the success of structure propagation, however, is highly dependent on the order in which the filling proceeds. We propose a best-first algorithm in which the confidence in the synthesized pixel values is propagated in a manner similar to the propagation of information in inpainting. The actual color values are computed using exemplar-based synthesis. In this paper, the simultaneous propagation of texture and structure information is achieved by a single , efficient algorithm. Computational efficiency is achieved by a block-based sampling process. A number of examples on real and synthetic images demonstrate the effectiveness of our algorithm in removing large occluding objects, as well as thin scratches. Robustness with respect to the shape of the manually selected target region is also demonstrated. Our results compare favorably to those obtained by existing techniques.

/pdf/region-filling-and-object-removal-by-exemplar-based-image-4pegss3e0m.pdf

Region filling and object removal by exemplar-based image inpainting

Criticism and the Growth of Knowledge. Edited by I. Lakatos and A. Musgrave. Pp. viii, 282. £3·50, paperback £1. 1970. (Cambridge University Press.)

Using generic interpolation machinery based on solving Poisson equations, a variety of novel tools are introduced for seamless editing of image regions. The first set of tools permits the seamless importation of both opaque and transparent source image regions into a destination region. The second set is based on similar mathematical ideas and allows the user to modify the appearance of the image seamlessly, within a selected region. These changes can be arranged to affect the texture, the illumination, and the color of objects lying in the region, or to make tileable a rectangular selection.

Poisson image editing

We present a novel approach for image completion that results in images that are both locally and globally consistent. With a fully-convolutional neural network, we can complete images of arbitrary resolutions by filling-in missing regions of any shape. To train this image completion network to be consistent, we use global and local context discriminators that are trained to distinguish real images from completed ones. The global discriminator looks at the entire image to assess if it is coherent as a whole, while the local discriminator looks only at a small area centered at the completed region to ensure the local consistency of the generated patches. The image completion network is then trained to fool the both context discriminator networks, which requires it to generate images that are indistinguishable from real ones with regard to overall consistency as well as in details. We show that our approach can be used to complete a wide variety of scenes. Furthermore, in contrast with the patch-based approaches such as PatchMatch, our approach can generate fragments that do not appear elsewhere in the image, which allows us to naturally complete the images of objects with familiar and highly specific structures, such as faces.

Globally and locally consistent image completion

A variational approach for filling-in regions of missing data in digital images is introduced. The approach is based on joint interpolation of the image gray levels and gradient/isophotes directions, smoothly extending in an automatic fashion the isophote lines into the holes of missing data. This interpolation is computed by solving the variational problem via its gradient descent flow, which leads to a set of coupled second order partial differential equations, one for the gray-levels and one for the gradient orientations. The process underlying this approach can be considered as an interpretation of the Gestaltist's principle of good continuation. No limitations are imposed on the topology of the holes, and all regions of missing data can be simultaneously processed, even if they are surrounded by completely different structures. Applications of this technique include the restoration of old photographs and removal of superimposed text like dates, subtitles, or publicity. Examples of these applications are given. We conclude the paper with a number of theoretical results on the proposed variational approach and its corresponding gradient descent flow.

/pdf/filling-in-by-joint-interpolation-of-vector-fields-and-gray-15yxs2b2kl.pdf

Filling-in by joint interpolation of vector fields and gray levels

We propose an algorithm to increase the resolution of multispectral satellite images knowing the panchromatic image at high resolution and the spectral channels at lower resolution. Our algorithm is based on the assumption that, to a large extent, the geometry of the spectral channels is contained in the topographic map of its panchromatic image. This assumption, together with the relation of the panchromatic image to the spectral channels, and the expression of the low-resolution pixel in terms of the high-resolution pixels given by some convolution kernel followed by subsampling, constitute the elements for constructing an energy functional (with several variants) whose minima will give the reconstructed spectral images at higher resolution. We discuss the validity of the above approach and describe our numerical procedure. Finally, some experiments on a set of multispectral satellite images are displayed.

/pdf/a-variational-model-for-p-xs-image-fusion-20x62i165j.pdf

A Variational Model for P+XS Image Fusion

Several explanations concerning notation, terminology, and background are in order. First notation: by 7Hi we have denoted the one-dimensional Hausdorff measure (i.e. length), and A(z,r) stands for the closed disc with center z and radius r. A curve F is called AD-regular, that is, Ahlfors-David-regular, if it satisfies (1.2) (with E = F). Since the lower bound is automatic for curves, this means that 'Hl (r n A(z, r)) 0. General sets satisfying (1.2) are called AD-regular. It is simplest to define the L2-boundedness of the Cauchy singular integral operator via the truncated integrals: we say that CE is bounded in L2(E) (without really defining the operator CE itself) if there is M < ox such that

The Cauchy integral, analytic capacity, and uniform rectifiability

In this paper we give a new proof of the L2 boundedness of the Cauchy integral on Lipschitz graphs (and chord-arc curves). Our method consists in controlling the Cauchy integral by an appropiate square function measuring the curvature of the graph. The square function is then estimated via a Fourier transform computation. Let Γ = {(x, y) ∈ R2 : y = A(x)} be the graph of a Lipschitz function A defined on the real line. Then A is locally absolutely continuous and A′ is bounded. The Cauchy integral of f ∈ L2(Γ ) is

A geometric proof of the L2 boundedness of the Cauchy integral on Lipschitz graphs

We show that the boundary of a rotating vortex patch (or V-state, in the terminology of Deem and Zabusky) is C∞, provided the patch is close to the bifurcation circle in the Lipschitz norm. The rotating patch is also convex if it is close to the bifurcation circle in the C2 norm. Our proof is based on Burbea’s approach to V-states.

Joan Verdera

Papers

Filling-in by joint interpolation of vector fields and gray levels

A Variational Model for P+XS Image Fusion

The Cauchy integral, analytic capacity, and uniform rectifiability

A geometric proof of the L2 boundedness of the Cauchy integral on Lipschitz graphs

Boundary Regularity of Rotating Vortex Patches