Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, we introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, we propose a two-faceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising. The source code and pre-trained models are available at https://github.com/swz30/MPRNet.

/pdf/multi-stage-progressive-image-restoration-3wfddniu8m.pdf

Multi-Stage Progressive Image Restoration

A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic

The current superresolution (SR) methods based on deep learning have shown remarkable comparative advantages but remain unsatisfactory in recovering the high-frequency edge details of the images in noise-contaminated imaging conditions, e.g., remote sensing satellite imaging. In this paper, we propose a generative adversarial network (GAN)-based edge-enhancement network (EEGAN) for robust satellite image SR reconstruction along with the adversarial learning strategy that is insensitive to noise. In particular, EEGAN consists of two main subnetworks: an ultradense subnetwork (UDSN) and an edge-enhancement subnetwork (EESN). In UDSN, a group of 2-D dense blocks is assembled for feature extraction and to obtain an intermediate high-resolution result that looks sharp but is eroded with artifacts and noises as previous GAN-based methods do. Then, EESN is constructed to extract and enhance the image contours by purifying the noise-contaminated components with mask processing. The recovered intermediate image and enhanced edges can be combined to generate the result that enjoys high credibility and clear contents. Extensive experiments on Kaggle Open Source Data set , Jilin-1 video satellite images, and Digitalglobe show superior reconstruction performance compared to the state-of-the-art SR approaches.

Edge-Enhanced GAN for Remote Sensing Image Superresolution

In order to effectively prevent the spread of COVID-19 virus, almost everyone wears a mask during coronavirus epidemic. This almost makes conventional facial recognition technology ineffective in many cases, such as community access control, face access control, facial attendance, facial security checks at train stations, etc. Therefore, it is very urgent to improve the recognition performance of the existing face recognition technology on the masked faces. Most current advanced face recognition approaches are designed based on deep learning, which depend on a large number of face samples. However, at present, there are no publicly available masked face recognition datasets. To this end, this work proposes three types of masked face datasets, including Masked Face Detection Dataset (MFDD), Real-world Masked Face Recognition Dataset (RMFRD) and Simulated Masked Face Recognition Dataset (SMFRD). Among them, to the best of our knowledge, RMFRD is currently theworld's largest real-world masked face dataset. These datasets are freely available to industry and academia, based on which various applications on masked faces can be developed. The multi-granularity masked face recognition model we developed achieves 95% accuracy, exceeding the results reported by the industry. Our datasets are available at: this https URL.

Masked Face Recognition Dataset and Application

In this paper, we explore the role of Instance Normalization in low-level vision tasks. Specifically, we present a novel block: Half Instance Normalization Block (HIN Block), to boost the performance of image restoration networks. Based on HIN Block, we design a simple and powerful multi-stage network named HINet, which consists of two subnetworks. With the help of HIN Block, HINet surpasses the state-of-the-art (SOTA) on various image restoration tasks. For image denoising, we exceed it 0.11dB and 0.28 dB in PSNR on SIDD dataset, with only 7.5% and 30% of its multiplier-accumulator operations (MACs), 6.8× and 2.9× speedup respectively. For image deblurring, we get comparable performance with 22.5% of its MACs and 3.3× speedup on REDS and GoPro datasets. For image deraining, we exceed it by 0.3 dB in PSNR on the average result of multiple datasets with 1.4× speedup. With HINet, we won the 1st place on the NTIRE 2021 Image Deblurring Challenge - Track2. JPEG Artifacts, with a PSNR of 29.70.

/pdf/hinet-half-instance-normalization-network-for-image-34473ablug.pdf

HINet: Half Instance Normalization Network for Image Restoration

Pedestrian trajectory prediction in multiple scenarios is of immense importance in autonomous driving and disentanglement of human behavior but is limited in catching human intention and initiative. Most previous works tend to predict the trajectory using only 2D coordinates, which generally cause two common problems: a) Overlooking the subjective initiative, including sudden swerve and erratic movement; b) A potential challenge called abnormal collision caused by unlabeled pedestrians on dataset is not being identified and resolved, which would ruin the model prediction. To break those limitations, we introduce visual localization and orientation as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Visual Intention Knowledge</i> to help the trajectory prediction, which is learned directly from visual scenarios. It benefits to comprehend human intention and formulates decision-making processes. Moreover, by learning from the visual information and decision-making policy, we construct the Visual Intention Knowledge associated spatio-temporal Transformer (VIKT) to predict human trajectory by combining the intention knowledge with the novel Transformer. Extensive experimental results demonstrate that our VIKT model could achieve competitive performance by the Visual Intention Knowledge through optimizing the model prediction compared with state-of-the-art methods in terms of prediction accuracy on ETH/UCY and SDD benchmarks. 

Visual Exposes You: Pedestrian Trajectory Prediction Meets Visual Intention

Spatial-Temporal Video Super-Resolution (ST-VSR) aims to generate high-quality videos with higher resolution (HR) and higher frame rate (HFR). Quite intuitively, pioneering two-stage based methods complete ST-VSR by directly combining two sub-tasks: Spatial Video Super-Resolution (S-VSR) and Temporal Video Super-Resolution (T-VSR) but ignore the reciprocal relations among them. 1) T-VSR to S-VSR: temporal correlations help accurate spatial detail representation; 2) S-VSR to T-VSR: abundant spatial information contributes to the refinement of temporal prediction. To this end, we propose a one-stage based Cycle-projected Mutual learning network (CycMuNet) for ST-VSR, which makes full use of spatial-temporal correlations via the mutual learning between S-VSR and T-VSR. Specifically, we propose to exploit the mutual information among them via iterative up- and down projections, where spatial and temporal features are fully fused and distilled, helping high-quality video reconstruction. In addition, we also show interesting extensions for efficient network design (CycMuNet+), such as parameter sharing and dense connection on projection units and feedback mechanism in CycMuNet. Besides extensive experiments on benchmark datasets, we also compare our proposed CycMuNet (+) with S-VSR and T-VSR tasks, demonstrating that our method significantly outperforms the state-of-the-art methods. Codes are publicly available at: https://github.com/hhhhhumengshun/CycMuNet.

CycMuNet+: Cycle-Projected Mutual Learning for Spatial-Temporal Video Super-Resolution.

Action recognition has attracted growing interest recently. It suffers from the problem that complex and diverse environments may disturb the extraction of action features. Existing methods propose to explore the temporal associations to alleviate the issue. However, they cannot handle long-range frames, and the rigid techniques are powerless against the differences caused by the deformation of the actors. To this end, we propose the Actor-Aware Alignment Network (A<inline-formula><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula>Net), which helps locate the action region. Specifically, through the intra-snippet correction, we afford the local segment alignment frames. The inter-snippet is designed to rectify the results, avoiding the occlusion situation that may appear in the local snippet. In addition, we consider intra-alignment short-range adjustive frames and long-range context frames between different snippets, which allows our A<inline-formula><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula>Net network to achieve the effect of focusing on long-range frame information. Multiple Reasoning Attention (MRA) modules are introduced to integrate features along the temporal dimension to keep the video spatio-temporal consistent. Extensive experiments conducted on three widely-used public benchmarks, <bold>UCF101</bold>, <bold>HMDB51</bold>, and <bold>InfAR</bold>, indicate that the excellence of our approach over other state-of-the-art models in wild scenarios.

Kui Jiang

Papers

Visual Exposes You: Pedestrian Trajectory Prediction Meets Visual Intention

CycMuNet+: Cycle-Projected Mutual Learning for Spatial-Temporal Video Super-Resolution.

Actor-Aware Alignment Network for Action Recognition