Local smoothness and nonlocal similarity have both led to sparsity prior useful to image recovery applications. In this paper, we propose to combine the strengths of local and nonlocal sparse representations by Bayesian model averaging (BMA) where sparsity offers a plausible approximation of model posterior probabilities. An iterative thresholding-based image recovery algorithm using hybrid sparse representations is developed and its convergence property is analyzed using the theory of fixed point. Since nonlocal sparsity based on clustering relationship is nonconvex, we have borrowed the powerful idea of deterministic annealing (DA) to optimize the algorithm performance. It can be shown that as temperature decreases, our algorithm is capable of traversing different states of image structures (e.g., smooth regions, regular edges and textures). Fully reproducible experimental results are reported to support the effectiveness of the proposed image recovery algorithm.

Image Recovery Via Hybrid Sparse Representations: A Deterministic Annealing Approach

The patch transform represents an image as a bag of overlapping patches sampled on a regular grid. This representation allows users to manipulate images in the patch domain, which then seeds the inverse patch transform to synthesize modified images. Possible modifications include the spatial locations of patches, the size of the output image, or the pool of patches from which an image is reconstructed. When no modifications are made, the inverse patch transform reduces to solving a jigsaw puzzle. The inverse patch transform is posed as a patch assignment problem on a Markov random field (MRF), where each patch should be used only once and neighboring patches should fit to form a plausible image. We find an approximate solution to the MRF using loopy belief propagation, introducing an approximation that encourages the solution to use each patch only once. The image reconstruction algorithm scales well with the total number of patches through label pruning. In addition, structural misalignment artifacts are suppressed through a patch jittering scheme that spatially jitters the assigned patches. We demonstrate the patch transform and its effectiveness on natural images.

The Patch Transform

Image inpainting based on deep learning: A review

Perhaps surprisingly sewerage infrastructure is one of the most costly infrastructures in modern society. Sewer pipes are manually inspected to determine whether the pipes are defective. However, this process is limited by the number of qualified inspectors and the time it takes to inspect a pipe. Automatization of this process is therefore of high interest. So far, the success of computer vision approaches for sewer defect classification has been limited when compared to the success in other fields mainly due to the lack of public datasets. To this end, in this work we present a large novel and publicly available multi-label classification dataset for image-based sewer defect classification called Sewer-ML.The Sewer-ML dataset consists of 1.3 million images annotated by professional sewer inspectors from three different utility companies across nine years. Together with the dataset, we also present a benchmark algorithm and a novel metric for assessing performance. The benchmark algorithm is a result of evaluating 12 state-of-the-art algorithms, six from the sewer defect classification domain and six from the multi-label classification domain, and combining the best performing algorithms. The novel metric is a class-importance weighted F2 score, F2 CIW , reflecting the economic impact of each class, used together with the normal pipe F1 score, F1 Normal . The benchmark algorithm achieves an F2 CIW score of 55.11% and F1 Normal score of 90.94%, leaving ample room for improvement on the Sewer-ML dataset. The code, models, and dataset are available at the project page http://vap.aau.dk/sewer-ml

/pdf/sewer-ml-a-multi-label-sewer-defect-classification-dataset-ndmw4td97c.pdf

Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark

Sewer defect detection from 3D point clouds using a transformer-based deep learning model

Automatic sewer pipe defect semantic segmentation based on improved U-Net

Structure-aware image completion has the reputation of keeping the salient structure of images, except for the case in texture propagation of images with distinguishing texture characteristics and large missing region. This inspirits us to design this new algorithm that could intensify texture-aware functions in the structure completion processing in an optimal manner. To avoid the occurrence of visually inconsistent results, we consider the filling of missing region as the energy minimization, using Belief Propagation (BP), of discrete Markov random field with integrated constraints of the costs of label, structure coherence and texture coherence. Moreover, wavelet multi-resolution image pyramid is adopted in our method, which not only enhances the global geometric structures and detailed texture features within large missing region, but as well speeds up the rate of convergence in the synthesis process.

Structure-Aware Image Completion with Texture Propagation

Video summarization aims to provide a compact video representation while preserving the essential activities of the original video. Most existing video summarization approaches relay on identifying important frames and optimizing target energy by a global optimum solution. But global optimum may fail to express continuous action or realistically validate how human beings perceive a story. In this paper, we present a bottom-up approach named clip growing for video summarization, which allows users to customize the quality of the video summaries. The proposed approach firstly uses clustering to oversegment video frames into video clips based on their similarity and proximity. Simultaneously, the importance of frames and clips is evaluated from their corresponding dissimilarity and representativeness. Then, video clips and frames are gradually selected according to their energy rank, until reaching the target length. Experimental results on SumMe dataset show that our algorithm can produce promising results compared to existing algorithms. Several video summarizations results are presented in supplementary material.

/pdf/a-bottom-up-summarization-algorithm-for-videos-in-the-wild-5co8kbc15y.pdf

A bottom-up summarization algorithm for videos in the wild

Face completion is a challenging task in computer vision. Unlike general images, face images usually have strong semantic correlation and symmetry. Without taking these characteristics into account, existing face completion techniques usually fail to produce a photo-realistic result, especially for the missing key components (e.g., eyes and mouths). In this paper, we propose a symmetry-aware face completion method based on facial structural features using a deep generative model. The model is trained with a combination of a reconstruction loss, a structure loss, two adversarial losses and a symmetry loss, which ensures pixel faithfulness, local-global contents integrity and symmetrical consistency. We conduct a dedicated symmetry detection technique for facial components and show that the symmetrical attention module significantly improves face completion results. Experiments show that our method is capable of synthesizing semantically valid and visually plausible contents for the missing facial key parts from random mask. In addition, our model outperforms other methods for detail completion of facial components.

Symmetry-Aware Face Completion with Generative Adversarial Networks

Plenty of multi-view video processing tasks such as video abstract, key-frame extraction and camera selection focus on presenting to audiences the most significant information in a certain period of time. Basically, the main idea of these techniques is to show audiences the video segments or frames that have the highest spatio-temporal significances. However, existing approaches are not enough to deal with these tasks with a general framework. In this paper, we develop a novel bottom-up algorithm called video clip growth that generates multi-view video abstract through an accurate frames adding process, which allows users to customize the length of the video summaries. This approach firstly uses an energy function to evaluate each frame’s importance from both time and space dimension. Then video clips and frames are gradually selected according to their energy rank, until reaching the target length. Besides, our algorithm can also extend to several multi-view video processing tasks. The experimental results on the Lobby and Office dataset have demonstrated the effectiveness of our algorithm.

Gang Pan

Papers

Automatic sewer pipe defect semantic segmentation based on improved U-Net

Structure-Aware Image Completion with Texture Propagation

A bottom-up summarization algorithm for videos in the wild

Symmetry-Aware Face Completion with Generative Adversarial Networks

Video Clip Growth: A General Algorithm for Multi-view Video Summarization