Theories of object recognition agree that shape is of primordial importance, but there is no consensus about how shape might be represented, and so far attempts to implement a model of shape perception that would work with realistic stimuli have largely failed. Recent studies suggest that state-of-the-art convolutional ‘deep’ neural networks (DNNs) capture important aspects of human object perception. We hypothesized that these successes might be partially related to a human-like representation of object shape. Here we demonstrate that sensitivity for shape features, characteristic to human and primate vision, emerges in DNNs when trained for generic object recognition from natural photographs. We show that these models explain human shape judgments for several benchmark behavioral and neural stimulus sets on which earlier models mostly failed. In particular, although never explicitly trained for such stimuli, DNNs develop acute sensitivity to minute variations in shape and to non-accidental properties that have long been implicated to form the basis for object recognition. Even more strikingly, when tested with a challenging stimulus set in which shape and category membership are dissociated, the most complex model architectures capture human shape sensitivity as well as some aspects of the category structure that emerges from human judgments. As a whole, these results indicate that convolutional neural networks not only learn physically correct representations of object categories but also develop perceptually accurate representational spaces of shapes. An even more complete model of human object representations might be in sight by training deep architectures for multiple tasks, which is so characteristic in human development.

/pdf/deep-neural-networks-as-a-computational-model-for-human-2ouij8cpt4.pdf

Deep Neural Networks as a Computational Model for Human Shape Sensitivity

AV1 is an emerging open-source and royalty-free video compression format, which is jointly developed and finalized in early 2018 by the Alliance for Open Media (AOMedia) industry consortium. The main goal of AV1 development is to achieve substantial compression gain over state-of-the-art codecs while maintaining practical decoding complexity and hardware feasibility. This paper provides a brief technical overview of key coding techniques in AV1 along with preliminary compression performance comparison against VP9 and HEVC.

An Overview of Core Coding Tools in the AV1 Video Codec

This paper presents CompressAI, a platform that provides custom operations, layers, models and tools to research, develop and evaluate end-to-end image and video compression codecs. In particular, CompressAI includes pre-trained models and evaluation tools to compare learned methods with traditional codecs. Multiple models from the state-of-the-art on learned end-to-end compression have thus been reimplemented in PyTorch and trained from scratch. We also report objective comparison results using PSNR and MS-SSIM metrics vs. bit-rate, using the Kodak image dataset as test set. Although this framework currently implements models for still-picture compression, it is intended to be soon extended to the video compression domain.

CompressAI: a PyTorch library and evaluation platform for end-to-end compression research.

/pdf/20shi-ji-noming-zhu-ming-lun-david-marr-vision-a-2y3j3ibp76.pdf

20世紀の名著名論：David Marr:Vision:a Computational Investigation into the Human Representation and Processing of Visual Information

In learning-based approaches to image compression, codecs are developed by optimizing a computational model to minimize a rate-distortion objective. Currently, the most effective learned image codecs take the form of an entropy-constrained autoencoder with an entropy model that uses both forward and backward adaptation. Forward adaptation makes use of side information and can be efficiently integrated into a deep neural network. In contrast, backward adaptation typically makes predictions based on the causal context of each symbol, which requires serial processing that prevents efficient GPU / TPU utilization. We introduce two enhancements, channel-conditioning and latent residual prediction, that lead to network architectures with better rate-distortion performance than existing context-adaptive models while minimizing serial processing. Empirically, we see an average rate savings of 6.7% on the Kodak image set and 11.4% on the Tecnick image set compared to a context-adaptive baseline model. At low bit rates, where the improvements are most effective, our model saves up to 18% over the baseline and outperforms hand-engineered codecs like BPG by up to 25%.

Channel-wise Autoregressive Entropy Models for Learned Image Compression

The AV1 video compression format is developed by the Alliance for Open Media consortium. It achieves more than a 30% reduction in bit rate compared to its predecessor VP9 for the same decoded video quality. This article provides a technical overview of the AV1 codec design that enables the compression performance gains with considerations for hardware feasibility.

/pdf/a-technical-overview-of-av1-3dxoopbuo8.pdf

A Technical Overview of AV1

In 2018, the Alliance for Open Media (AOMedia) finalized its first video compression format AV1, which is jointly developed by the industry consortium of leading video technology companies. The main goal of AV1 is to provide an open source and royalty-free video coding format that substantially outperforms state-of-the-art codecs available on the market in compression efficiency while remaining practical decoding complexity as well as being optimized for hardware feasibility and scalability on modern devices. To give detailed insights into how the targeted performance and feasibility is realized, this paper provides a technical overview of key coding techniques in AV1. Besides, the coding performance gains are validated by video compression tests performed with the libaom AV1 encoder against the libvpx VP9 encoder. Preliminary comparison with two leading HEVC encoders, x265 and HM, and the reference software of VVC is also conducted on AOM's common test set and an open 4k set.

/pdf/an-overview-of-coding-tools-in-av1-the-first-video-codec-32runghbs7.pdf

An Overview of Coding Tools in AV1: the First Video Codec from the Alliance for Open Media

Image restoration schemes have traditionally been targeted only for use in a blind scenario, where the aim is to improve the quality of an image or video after it has suffered degradations in capture, processing, storage, coding or transmission, at a time when the undegraded source is no longer available. However, these schemes can also be adopted for compression, because any information that can be restored can be saved, thereby aiding compressibility. In this use case, the source is known at the time of compression, and the encoder can send additional side-information to the decoder with the compressed bit-stream to specify how the decoder is supposed to restore. In this paper, we describe three novel schemes that are applied in-loop to reconstructed frames after a conventional deblocking loop filter has been applied. These schemes are switchable within a frame per suitably sized tile. The specific schemes described are based on separable symmetric Wiener filters, dual self-guided filters with subspace projection, and domain transform recursive filters. Results with the new coding tool are presented on the emerging AV1 video codec on standard test sets, showing upwards of 2% bit-rate savings.

A switchable loop-restoration with side-information framework for the emerging AV1 video codec

Real motion is often complex, and cannot always be sufficiently described by a translational model for the purpose of video coding. Translational motion models can only map a rectangular to a rectangular of the same size in a reference frame, but real motion observed in videos often go beyond that. For example, motion due to camera shake, panning and zoom might require transformations that support shearing, scaling, rotation and changes in aspect ratio. In order to handle these instances adequately for video encoding, we introduce two coding modes — a global motion mode and a locally adaptive warped motion mode, that make use of affine and/or homographic projections to more accurately describe and predict complex motion. These warped modes respectively capture global motion computed at the frame level, or local motion computed at the block level based on local motion statistics. We pay special attention to efficient SIMD friendly decoder-side warping operations to make these tools practically useful. Together, these techniques prove to yield considerable coding gains on standard test sets.

Sarah Parker

Papers

An Overview of Core Coding Tools in the AV1 Video Codec

A Technical Overview of AV1

An Overview of Coding Tools in AV1: the First Video Codec from the Alliance for Open Media

A switchable loop-restoration with side-information framework for the emerging AV1 video codec

Global and locally adaptive warped motion compensation in video compression