Mathematical models for local nontexture inpaintings
01 Jan 2002-Siam Journal on Applied Mathematics (Society for Industrial and Applied Mathematics)-Vol. 62, Iss: 3, pp 1019-1043
TL;DR: The broad applications of the inpainting models are demonstrated through restoring scratched old photos, disocclusion in vision analysis, text removal, digital zooming, and edge-based image coding.
Abstract: Dedicated to Stanley Osher on the occasion of his 60th birthday. Abstract. Inspired by the recent work of Bertalmio et al. on digital inpaintings (SIGGRAPH 2000), we develop general mathematical models for local inpaintings of nontexture images. On smooth regions, inpaintings are connected to the harmonic and biharmonic extensions, and inpainting orders are analyzed. For inpaintings involving the recovery of edges, we study a variational model that is closely connected to the classical total variation (TV) denoising model of Rudin, Osher, and Fatemi (Phys. D, 60 (1992), pp. 259-268). Other models are also discussed based on the Mumford-Shah regularity (Comm. Pure Appl. Math., XLII (1989), pp. 577-685) and curvature driven diffusions (CDD) of Chan and Shen (J. Visual Comm. Image Rep., 12 (2001)). The broad applications of the inpainting models are demonstrated through restoring scratched old photos, disocclusion in vision analysis, text removal, digital zooming, and edge-based image coding.
Citations
More filters
TL;DR: It is shown that CLBP_S preserves more information of the local structure thanCLBP_M, which explains why the simple LBP operator can extract the texture features reasonably well and can be made for rotation invariant texture classification.
Abstract: In this correspondence, a completed modeling of the local binary pattern (LBP) operator is proposed and an associated completed LBP (CLBP) scheme is developed for texture classification. A local region is represented by its center pixel and a local difference sign-magnitude transform (LDSMT). The center pixels represent the image gray level and they are converted into a binary code, namely CLBP-Center (CLBP_C), by global thresholding. LDSMT decomposes the image local differences into two complementary components: the signs and the magnitudes, and two operators, namely CLBP-Sign (CLBP_S) and CLBP-Magnitude (CLBP_M), are proposed to code them. The traditional LBP is equivalent to the CLBP_S part of CLBP, and we show that CLBP_S preserves more information of the local structure than CLBP_M, which explains why the simple LBP operator can extract the texture features reasonably well. By combining CLBP_S, CLBP_M, and CLBP_C features into joint or hybrid distributions, significant improvement can be made for rotation invariant texture classification.
1,981 citations
TL;DR: This topic can be viewed as an extension of spectral graph theory and the diffusion geometry framework to functional analysis and PDE-like evolutions to define new types of flows and functionals for image processing and elsewhere.
Abstract: We propose the use of nonlocal operators to define new types of flows and functionals for image processing and elsewhere. A main advantage over classical PDE-based algorithms is the ability to handle better textures and repetitive structures. This topic can be viewed as an extension of spectral graph theory and the diffusion geometry framework to functional analysis and PDE-like evolutions. Some possible applications and numerical examples are given, as is a general framework for approximating Hamilton–Jacobi equations on arbitrary grids in high demensions, e.g., for control theory.
1,397 citations
TL;DR: A functional with variable exponent, which provides a model for image denoising, enhancement, and restoration, is studied and the existence, uniqueness, and long-time behavior of the proposed model are established.
Abstract: We study a functional with variable exponent, $1\leq p(x)\leq 2$, which provides a model for image denoising, enhancement, and restoration. The diffusion resulting from the proposed model is a combination of total variation (TV)-based regularization and Gaussian smoothing. The existence, uniqueness, and long-time behavior of the proposed model are established. Experimental results illustrate the effectiveness of the model in image restoration.
1,328 citations
Cites background from "Mathematical models for local nonte..."
...6) also has direct application in image processing, as it can be used for image interpolation [13, 27], also referred to as noise-free image inpainting [8, 17, 18]....
[...]
01 Jul 2017
TL;DR: A novel method for semantic image inpainting, which generates the missing content by conditioning on the available data, and successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.
Abstract: Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.
1,258 citations
Cites background from "Mathematical models for local nonte..."
...For example, total variation (TV) based approaches [34, 1] take into account the smoothness property of natural images, which is useful to fill small missing regions or remove spurious noise....
[...]
TL;DR: The novel contribution of this paper is the combination of these three previously developed components, image decomposition with inpainting and texture synthesis, which permits the simultaneous use of filling-in algorithms that are suited for different image characteristics.
Abstract: An algorithm for the simultaneous filling-in of texture and structure in regions of missing image information is presented in this paper. The basic idea is to first decompose the image into the sum of two functions with different basic characteristics, and then reconstruct each one of these functions separately with structure and texture filling-in algorithms. The first function used in the decomposition is of bounded variation, representing the underlying image structure, while the second function captures the texture and possible noise. The region of missing information in the bounded variation image is reconstructed using image inpainting algorithms, while the same region in the texture image is filled-in with texture synthesis techniques. The original image is then reconstructed adding back these two sub-images. The novel contribution of this paper is then in the combination of these three previously developed components, image decomposition with inpainting and texture synthesis, which permits the simultaneous use of filling-in algorithms that are suited for different image characteristics. Examples on real images show the advantages of this proposed approach.
1,024 citations
References
More filters
[...]
40,330 citations
"Mathematical models for local nonte..." refers methods in this paper
...Inspired by the recent work of Bertalmio, Sapiro, Caselles, and Ballester [Technicalreport, ECE-University of Minnesota (1999)] on digital inpaintings, we develop general mathemat-ical models for local non-texture inpaintings....
[...]
...[4] V. Caselles, J.-M. Morel, and C. Sbert....
[...]
...REFERENCES[1] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester....
[...]
...[43] B. Tang, G. Sapiro, and V. Caselles....
[...]
...New applications of the TV model forrestoring non- at image features such as optical ows and chromaticity can be foundin the recent papers by Perona [36], Tang, Sapiro, and Caselles [42, 43], and Chanand Shen [9]....
[...]
TL;DR: The analogy between images and statistical mechanics systems is made and the analogous operation under the posterior distribution yields the maximum a posteriori (MAP) estimate of the image given the degraded observations, creating a highly parallel ``relaxation'' algorithm for MAP estimation.
Abstract: We make an analogy between images and statistical mechanics systems. Pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice-like physical system. The assignment of an energy function in the physical system determines its Gibbs distribution. Because of the Gibbs distribution, Markov random field (MRF) equivalence, this assignment also determines an MRF image model. The energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the MRF. For a range of degradation mechanisms, including blurring, nonlinear deformations, and multiplicative or additive noise, the posterior distribution is an MRF with a structure akin to the image model. By the analogy, the posterior distribution defines another (imaginary) physical system. Gradual temperature reduction in the physical system isolates low energy states (``annealing''), or what is the same thing, the most probable states under the Gibbs distribution. The analogous operation under the posterior distribution yields the maximum a posteriori (MAP) estimate of the image given the degraded observations. The result is a highly parallel ``relaxation'' algorithm for MAP estimation. We establish convergence properties of the algorithm and we experiment with some simple pictures, for which good restorations are obtained at low signal-to-noise ratios.
18,761 citations
"Mathematical models for local nonte..." refers background in this paper
...6 Chan and Shen the world (see, for example, the well-known paper by the Geman brothers [16])....
[...]
Book•
07 Jan 2013
TL;DR: In this article, Leray-Schauder and Harnack this article considered the Dirichlet Problem for Poisson's Equation and showed that it is a special case of Divergence Form Operators.
Abstract: Chapter 1. Introduction Part I: Linear Equations Chapter 2. Laplace's Equation 2.1 The Mean Value Inequalities 2.2 Maximum and Minimum Principle 2.3 The Harnack Inequality 2.4 Green's Representation 2.5 The Poisson Integral 2.6 Convergence Theorems 2.7 Interior Estimates of Derivatives 2.8 The Dirichlet Problem the Method of Subharmonic Functions 2.9 Capacity Problems Chapter 3. The Classical Maximum Principle 3.1 The Weak Maximum Principle 3.2 The Strong Maximum Principle 3.3 Apriori Bounds 3.4 Gradient Estimates for Poisson's Equation 3.5 A Harnack Inequality 3.6 Operators in Divergence Form Notes Problems Chapter 4. Poisson's Equation and Newtonian Potential 4.1 Holder Continuity 4.2 The Dirichlet Problem for Poisson's Equation 4.3 Holder Estimates for the Second Derivatives 4.4 Estimates at the Boundary 4.5 Holder Estimates for the First Derivatives Notes Problems Chapter 5. Banach and Hilbert Spaces 5.1 The Contraction Mapping 5.2 The Method of Cintinuity 5.3 The Fredholm Alternative 5.4 Dual Spaces and Adjoints 5.5 Hilbert Spaces 5.6 The Projection Theorem 5.7 The Riesz Representation Theorem 5.8 The Lax-Milgram Theorem 5.9 The Fredholm Alternative in Hilbert Spaces 5.10 Weak Compactness Notes Problems Chapter 6. Classical Solutions the Schauder Approach 6.1 The Schauder Interior Estimates 6.2 Boundary and Global Estimates 6.3 The Dirichlet Problem 6.4 Interior and Boundary Regularity 6.5 An Alternative Approach 6.6 Non-Uniformly Elliptic Equations 6.7 Other Boundary Conditions the Obliue Derivative Problem 6.8 Appendix 1: Interpolation Inequalities 6.9 Appendix 2: Extension Lemmas Notes Problems Chapter 7. Sobolev Spaces 7.1 L^p spaces 7.2 Regularization and Approximation by Smooth Functions 7.3 Weak Derivatives 7.4 The Chain Rule 7.5 The W^(k,p) Spaces 7.6 DensityTheorems 7.7 Imbedding Theorems 7.8 Potential Estimates and Imbedding Theorems 7.9 The Morrey and John-Nirenberg Estimes 7.10 Compactness Results 7.11 Difference Quotients 7.12 Extension and Interpolation Notes Problems Chapter 8 Generalized Solutions and Regularity 8.1 The Weak Maximum Principle 8.2 Solvability of the Dirichlet Problem 8.3 Diferentiability of Weak Solutions 8.4 Global Regularity 8.5 Global Boundedness of Weak Solutions 8.6 Local Properties of Weak Solutions 8.7 The Strong Maximum Principle 8.8 The Harnack Inequality 8.9 Holder Continuity 8.10 Local Estimates at the Boundary 8.11 Holder Estimates for the First Derivatives 8.12 The Eigenvalue Problem Notes Problems Chapter 9. Strong Solutions 9.1 Maximum Princiles for Strong Solutions 9.2 L^p Estimates: Preliminary Analysis 9.3 The Marcinkiewicz Interpolation Theorem 9.4 The Calderon-Zygmund Inequality 9.5 L^p Estimates 9.6 The Dirichlet Problem 9.7 A Local Maximum Principle 9.8 Holder and Harnack Estimates 9.9 Local Estimates at the Boundary Notes Problems Part II: Quasilinear Equations Chapter 10. Maximum and Comparison Principles 10.1 The Comparison Principle 10.2 Maximum Principles 10.3 A Counterexample 10.4 Comparison Principles for Divergence Form Operators 10.5 Maximum Principles for Divergence Form Operators Notes Problems Chapter 11. Topological Fixed Point Theorems and Their Application 11.1 The Schauder Fixes Point Theorem 11.2 The Leray-Schauder Theorem: a Special Case 11.3 An Application 11.4 The Leray-Schauder Fixed Point Theorem 11.5 Variational Problems Notes Chapter 12. Equations in Two Variables 12.1 Quasiconformal Mappings 12.2 holder Gradient Estimates for Linear Equations 12.3 The Dirichlet Problem for Uniformly Elliptic Equations 12.4 Non-Uniformly Elliptic Equations Notes Problems Chapter 13. Holder Estimates for
18,443 citations
"Mathematical models for local nonte..." refers background in this paper
...0 for all z ∈ D1 due to the maximum principle of harmonic functions: The minimum is always achieved along the boundary (Gilbarg and Trudinger [19])....
[...]
Book•
01 May 1992TL;DR: This paper presents a meta-analyses of the wavelet transforms of Coxeter’s inequality and its applications to multiresolutional analysis and orthonormal bases.
Abstract: Introduction Preliminaries and notation The what, why, and how of wavelets The continuous wavelet transform Discrete wavelet transforms: Frames Time-frequency density and orthonormal bases Orthonormal bases of wavelets and multiresolutional analysis Orthonormal bases of compactly supported wavelets More about the regularity of compactly supported wavelets Symmetry for compactly supported wavelet bases Characterization of functional spaces by means of wavelets Generalizations and tricks for orthonormal wavelet bases References Indexes.
16,073 citations
"Mathematical models for local nonte..." refers background in this paper
...idea is very close to the multiresolution synthesis in wavelet decomposition or coding [ 14 , 44].) Let u∆ D be any linear inpainting of ∆u 0 D (via the harmonic scheme, for example)....
[...]
...Zoomout is a process of losing details or, in the framework of wavelets and multiresolution analysis, a process of projections from fine scales to coarser ones [ 14 , 44]....
[...]
TL;DR: In this article, a constrained optimization type of numerical algorithm for removing noise from images is presented, where the total variation of the image is minimized subject to constraints involving the statistics of the noise.
15,225 citations
"Mathematical models for local nonte..." refers background or methods in this paper
...In this spirit, the total variation (TV) inpainting model is formulated in Section 6, which extends the classical TV restoration model of Rudin, Osher and Fatemi [38]....
[...]
...In the scale-space theorybased on anisotropic di usions, images are considered as in the functional space ofbounded variations (Rudin, Osher, and Fatemi [38], and Chambolle and Lions [5])....
[...]
...4) for all z = (x; y) 2 E [ D, plus the Neumann boundary condition [9, 38]....
[...]
...In the scale-space theory based on anisotropic di usions, images are considered as in the functional space of bounded variations (Rudin, Osher, and Fatemi [38], and Chambolle and Lions [5])....
[...]
...It leads to the well known total variation (TV) restoration model of Rudin, Osher and Fatemi [38], where r(s) is taken to be s exactly....
[...]