There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

第12次 아시아ㆍ太平洋 矯政局長會議 參加記

The current superresolution (SR) methods based on deep learning have shown remarkable comparative advantages but remain unsatisfactory in recovering the high-frequency edge details of the images in noise-contaminated imaging conditions, e.g., remote sensing satellite imaging. In this paper, we propose a generative adversarial network (GAN)-based edge-enhancement network (EEGAN) for robust satellite image SR reconstruction along with the adversarial learning strategy that is insensitive to noise. In particular, EEGAN consists of two main subnetworks: an ultradense subnetwork (UDSN) and an edge-enhancement subnetwork (EESN). In UDSN, a group of 2-D dense blocks is assembled for feature extraction and to obtain an intermediate high-resolution result that looks sharp but is eroded with artifacts and noises as previous GAN-based methods do. Then, EESN is constructed to extract and enhance the image contours by purifying the noise-contaminated components with mask processing. The recovered intermediate image and enhanced edges can be combined to generate the result that enjoys high credibility and clear contents. Extensive experiments on Kaggle Open Source Data set , Jilin-1 video satellite images, and Digitalglobe show superior reconstruction performance compared to the state-of-the-art SR approaches.

Edge-Enhanced GAN for Remote Sensing Image Superresolution

In recent years, the image and video coding technologies have advanced by leaps and bounds. However, due to the popularization of image and video acquisition devices, the growth rate of image and video data is far beyond the improvement of the compression ratio. In particular, it has been widely recognized that there are increasing challenges of pursuing further coding performance improvement within the traditional hybrid coding framework. Deep convolution neural network which makes the neural network resurge in recent years and has achieved great success in both artificial intelligent and signal processing fields, also provides a novel and promising solution for image and video compression. In this paper, we provide a systematic, comprehensive and up-to-date review of neural network-based image and video compression techniques. The evolution and development of neural network-based compression methodologies are introduced for images and video respectively. More specifically, the cutting-edge video coding techniques by leveraging deep learning and HEVC framework are presented and discussed, which promote the state-of-the-art video coding performance substantially. Moreover, the end-to-end image and video coding frameworks based on neural networks are also reviewed, revealing interesting explorations on next generation image and video coding frameworks/standards. The most significant research works on the image and video coding related topics using neural networks are highlighted, and future trends are also envisioned. In particular, the joint compression on semantic and visual information is tentatively explored to formulate high efficiency signal representation structure for both human vision and machine vision, which are the two dominant signal receptors in the age of artificial intelligence.

Image and Video Compression With Neural Networks: A Review

Lightweight video compression has effectively alleviated the tension between growing transmission demands and expensive integration upgrades. Effective rate control algorithms are believed to be the crucial bottleneck for quality improvement during those ultra-high throughput coding processes. This paper proposes a novel rate control (RC) scheme that constructs a contextual adaptive bit estimation model through clustering historical compression information into block-gradient complexity categories. A buffer-aware tuning method and a flexible quantization parameter (QP) mapping algorithm are designed to determine the Luma/Chroma QP distribution where a simplified Lagrangian multiplier is further defined to preserve the stability of the overall compression process. As a result, the constant-bitrate compression towards low-latency interoperable ASICs is implemented with a promising RC performance.

An Efficient Rate Control Scheme for Video Compression in Low-latency Interoperable Interfaces

Depth compression plays an important role in 3D video coding with the typical texture-plus-depth representation. In this paper, to reduce the encoding complexity of depth map, we propose a low complexity adaptive View Synthesis Optimization scheme for the 3D extension of high efficiency video coding (3D-HEVC) standard. More specifically, we distinguish the coding tree units (CTUs) based on the influence of depth map compression on the quality of rendered synthesized view, and classify them into two categories, including synthesized view distortion change (SVDC) based and view synthesis distortion estimation (VSDE) based CTUs. In this manner, we can dynamically distinguish the CTUs and apply different rate-distortion optimization strategies. Moreover, for VSDE based CTUs, a new distortion model is proposed to infer the distortion of the synthesized view based on the depth distortion and texture characteristics. As such, we can achieve a good trade-off between the rate-distortion performance and computational complexity for depth map coding. Experimental results also confirm that the proposed scheme is effective in reducing the encoding complexity with ignorable rate-distortion performance loss compared with the state-of-the-art scheme in the 3D-HEVC platform.

Adaptive view synthesis optimization for low complexity 3D-HEVC encoding

Cross-component linear model (CCLM) prediction has been repeatedly proven to be effective in reducing the inter-channel redundancies in video compression. Essentially speaking, the linear model is identically trained by employing accessible luma and chroma reference samples at both encoder and decoder, elevating the level of operational complexity due to the least square regression or max-min based model parameter derivation. In this paper, we investigate the capability of the linear model in the context of sub-sampled based cross-component correlation mining, as a means of significantly releasing the operation burden and facilitating the hardware and software design for both encoder and decoder. In particular, the sub-sampling ratios and positions are elaborately designed by exploiting the spatial correlation and the inter-channel correlation. Extensive experiments verify that the proposed method is characterized by its simplicity in operation and robustness in terms of rate-distortion performance, leading to the adoption by Versatile Video Coding (VVC) standard and the third generation of Audio Video Coding Standard (AVS3).

Sub-sampled Cross-component Prediction for Emerging Video Coding Standards

Rate control plays an essential role in real video applications due to the limitation of the bandwidth and storage budget. Supported by reference picture resampling (RPR) in Versatile Video Coding (VVC), which enables resolution changes without introducing an intra-coded frame, it is possible to adjust picture resolutions to adapt to the required bit-rate. However, few rate control algorithms with resolution changes have been explored for VVC. Therefore, we propose a $\lambda$-domain rate control algorithm with resolution changes in this paper. In the proposed rate control scheme, the coding resolution of one frame is determined based on the property of the frame. To make the rate control scheme adaptive to mixed resolutions, the rate control parameters of the full resolution and the low resolution are updated separately. Furthermore, $\lambda$ and QP of the frames with different resolutions are unified to obtain appropriate coding parameters. Simulation results show that a BD-rate coding gain about 3.7% on average can be achieved with the reduction of computational complexity. Besides, negligible bitrate error can be observed at both sequence and frame level.

Rate Control with Resolution Changes for VVC

AbstractIn real-world scenarios, video analysis algorithms are conducted for visual signals after compression and transmission. Generally speaking, most codecs introduce irreversible distortion due to coarse quantization during compression. The distortion may lead to significant perception degradation in terms of video analysis performance. To tackle this problem, we propose an efficient plug-and-play approach to preserve the essential semantic information in video sequences explicitly. The proposed approach could boost the video analysis performance with a little extra bit cost. Specifically, we employ the proposed approach on an emerging video analysis task, video object segmentation (VOS). Massive experimental results prove that the our work outperforms the existing coding approaches over multiple VOS datasets. Concretely, it could improve the analysis performance by up to 13% at similar bitrates. Additional experiments also verifies the flexibility of our scheme because there is no dependency on any specific VOS model or encoding method. Essentially, the proposed approach provides novel insights for the emerging Video Coding for Machine (VCM) standard. 

Shanshe Wang

Papers

An Efficient Rate Control Scheme for Video Compression in Low-latency Interoperable Interfaces

Adaptive view synthesis optimization for low complexity 3D-HEVC encoding

Sub-sampled Cross-component Prediction for Emerging Video Coding Standards

Rate Control with Resolution Changes for VVC

A Compressive Prior Guided Mask Predictive Coding Approach for Video Analysis