Non-local methods for image denoising and inpainting have gained considerable attention in recent years. This is in part due to their superior performance in textured images, a known weakness of purely local methods. Local methods on the other hand have demonstrated to be very appropriate for the recovering of geometric structures such as image edges. The synthesis of both types of methods is a trend in current research. Variational analysis in particular is an appropriate tool for a unified treatment of local and non-local methods. In this work we propose a general variational framework for non-local image inpainting, from which important and representative previous inpainting schemes can be derived, in addition to leading to novel ones. We explicitly study some of these, relating them to previous work and showing results on synthetic and real images.

/pdf/a-variational-framework-for-exemplar-based-image-inpainting-49k6nnclrv.pdf

A Variational Framework for Exemplar-Based Image Inpainting

In this paper, we first introduce a general approach for context-aware patch-based image inpainting, where textural descriptors are used to guide and accelerate the search for well-matching (candidate) patches. A novel top-down splitting procedure divides the image into variable size blocks according to their context, constraining thereby the search for candidate patches to nonlocal image regions with matching context. This approach can be employed to improve the speed and performance of virtually any (patch-based) inpainting method. We apply this approach to the so-called global image inpainting with the Markov random field (MRF) prior, where MRF encodes a priori knowledge about consistency of neighboring image patches. We solve the resulting optimization problem with an efficient low-complexity inference method. Experimental results demonstrate the potential of the proposed approach in inpainting applications like scratch, text, and object removal. Improvement and significant acceleration of a related global MRF-based inpainting method is also evident.

Context-Aware Patch-Based Image Inpainting Using Markov Random Field Modeling

We present a novel formulation of exemplar-based inpainting as a global energy optimization problem, written in terms of the offset map. The proposed energy function combines a data attachment term that ensures the continuity of reconstruction at the boundary of the inpainting domain with a smoothness term that ensures a visually coherent reconstruction inside the hole. This formulation is adapted to obtain a global minimum using the graph cuts algorithm. To reduce the computational complexity, we propose an efficient multiscale graph cuts algorithm. To compensate the loss of information at low resolution levels, we use a feature representation computed at the original image resolution. This permits alleviation of the ambiguity induced by comparing only color information when the image is represented at low resolution levels. Our experiments show how well the proposed algorithm performs compared with other recent algorithms.

Exemplar-Based Image Inpainting Using Multiscale Graph Cuts

In this paper, we propose a novel inpainting algorithm combining the advantages of PDE-based schemes and examplar-based approaches. The proposed algorithm relies on the use of structure tensors to define the filling order priority and template matching. The structure tensors are computed in a hierarchic manner whereas the template matching is based on a K-nearest neighbor algorithm. The value K is adaptively set in function of the local texture information. Compared to two state of the art approaches, the proposed method provides more coherent results.

/pdf/examplar-based-inpainting-based-on-local-geometry-1m0co2kgr7.pdf

Examplar-based inpainting based on local geometry

Exemplar-based methods have proven their efficiency for the reconstruction of missing parts in a digital image. Texture as well as local geometry are often very well restored by such methods. Some applications, however, require the ability to reconstruct nonlocal geometric features, e.g., long edges. In order to do so, we propose to first compute a geometric sketch, which is then interpolated and used as a guide for the global reconstruction. In comparison with other related approaches, the originality of our work relies on the following points: (1) The geometric sketch computation is parameter-free and based on level lines, which provides a complete, reliable, and stable representation of the image. (2) The completion of the geometric sketch is fully automatic. It is done using a new—and interesting on its own—geometric inpainting approach that interpolates level lines with Euler spirals. Euler spirals are natural curves for shape completion and have been used already for edge completion and inpainting. It is the first time, however, that these curves are used for completing the whole level lines structure. (3) The general reconstruction is performed using a guided version of a classical exemplar-based method. However, we do not constrain the exemplar-based reconstruction to strictly follow the geometric guide. We actually use a new metric between blocks that consists of the sum of the classical ${{\mathrm L}^2}$ metric between any two blocks of the general image plus an ${{\mathrm L}^2}$ metric between the corresponding blocks in the completed geometric image. This is equivalent to a Lagrangian relaxation of a strictly guided reconstruction. We discuss in the paper the details of the method and some related mathematical issues, and we illustrate its efficiency on several examples.

https://hal.archives-ouvertes.fr/hal-00392018/document

Geometrically Guided Exemplar-Based Inpainting

Image inpainting techniques have been widely investigated to remove undesired objects in an image. Conventionally, missing parts in an image are completed by optimizing the objective function using pattern similarity. However, unnatural textures are easily generated due to two factors: (1) available samples in the image are quite limited, and (2) pattern similarity is one of the required conditions but is not sufficient for reproducing natural textures. In this paper, we propose a new energy function based on the pattern similarity considering brightness changes of sample textures (for (1)) and introducing spatial locality as an additional constraint (for (2)). The effectiveness of the proposed method is successfully demonstrated by qualitative and quantitative evaluation. Furthermore, the evaluation methods used in much inpainting research are discussed.

/pdf/image-inpainting-considering-brightness-change-and-spatial-jq8eyoo3xf.pdf

Image Inpainting Considering Brightness Change and Spatial Locality of Textures and Its Evaluation

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in real time. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

Diminished Reality Based on Image Inpainting Considering Background Geometry

This paper proposes a new diminished reality method for 3D scenes considering background structures. Most conventional methods using image inpainting assumes that the background around a target object is almost planar. In this study, approximating the background structure by the combination of local planes, perspective distortion of texture is corrected and searching area is limited for improving the quality of image inpainting. The temporal coherence of texture is preserved using the estimated structures and camera pose estimated by Visual-SLAM.

/pdf/diminished-reality-considering-background-structures-37n5vy9v1w.pdf

Diminished reality considering background structures

This paper proposes a new diminished reality technique which removes AR markers from a user's view image. In order to achieve natural marker hiding, three factors should be considered; (1) naturalness of texture generated on a marker area. (2) geometric consistency between consecutive frames, (3) photometric consistency between a marker area and its surrounding. In this study, assuming that an area around a marker is locally planar, the marker area in the first frame image is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in subsequent frames according to camera pose for temporal geometric consistency. Global and local luminance changes around the marker are reflected to the inpainted texture for photometric consistency.

/pdf/paper-diminished-reality-for-ar-marker-hiding-based-on-image-40hvquigoi.pdf

[Paper] Diminished Reality for AR Marker Hiding Based on Image Inpainting with Reflection of Luminance Changes

This paper proposes a novel image inpainting method to remove undesired objects in an image. Conventionally, missing regions are filled in using similar textures in an image as exemplars. However, unnatural textures are often generated due to the paucity of available samples. In this study, we take into account symmetric transformation of texture patterns to increase exemplars. To generate plausible textures in missing regions with variously transformed patterns, we employ two approaches: (1) we use spatial coherence of texture patterns when searching for similar patterns, and (2) we define a new degree of confidence of exemplars for determining pixel values. The effectiveness of the proposed method is demonstrated by comparing results by three methods.

/pdf/image-inpainting-considering-symmetric-patterns-3xvryrdmv6.pdf

Norihiko Kawai

Papers

Image Inpainting Considering Brightness Change and Spatial Locality of Textures and Its Evaluation

Diminished Reality Based on Image Inpainting Considering Background Geometry

Diminished reality considering background structures

[Paper] Diminished Reality for AR Marker Hiding Based on Image Inpainting with Reflection of Luminance Changes

Image inpainting considering symmetric patterns