“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

Knowledge Distillation: A Survey

Domain-specific hardware is becoming a promising topic in the backdrop of improvement slow down for general-purpose processors due to the foreseeable end of Moore’s Law. Machine learning, especially deep neural networks (DNNs), has become the most dazzling domain witnessing successful applications in a wide spectrum of artificial intelligence (AI) tasks. The incomparable accuracy of DNNs is achieved by paying the cost of hungry memory consumption and high computational complexity, which greatly impedes their deployment in embedded systems. Therefore, the DNN compression concept was naturally proposed and widely used for memory saving and compute acceleration. In the past few years, a tremendous number of compression techniques have sprung up to pursue a satisfactory tradeoff between processing efficiency and application accuracy. Recently, this wave has spread to the design of neural network accelerators for gaining extremely high performance. However, the amount of related works is incredibly huge and the reported approaches are quite divergent. This research chaos motivates us to provide a comprehensive survey on the recent advances toward the goal of efficient compression and execution of DNNs without significantly compromising accuracy, involving both the high-level algorithms and their applications in hardware design. In this article, we review the mainstream compression approaches such as compact model, tensor decomposition, data quantization, and network sparsification. We explain their compression principles, evaluation metrics, sensitivity analysis, and joint-way use. Then, we answer the question of how to leverage these methods in the design of neural network accelerators and present the state-of-the-art hardware architectures. In the end, we discuss several existing issues such as fair comparison, testing workloads, automatic compression, influence on security, and framework/hardware-level support, and give promising topics in this field and the possible challenges as well. This article attempts to enable readers to quickly build up a big picture of neural network compression and acceleration, clearly evaluate various methods, and confidently get started in the right way.

Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey

Since the early 1980s, most electronics have relied on the use of complementary metal–oxide–semiconductor (CMOS) transistors. However, the principles of CMOS operation, involving a switchable semiconductor conductance controlled by an insulating gate, have remained largely unchanged, even as transistors are miniaturized to sizes of 10 nanometres. We investigated what dimensionally scalable logic technology beyond CMOS could provide improvements in efficiency and performance for von Neumann architectures and enable growth in emerging computing such as artifical intelligence. Such a computing technology needs to allow progressive miniaturization, reduce switching energy, improve device interconnection and provide a complete logic and memory family. Here we propose a scalable spintronic logic device that operates via spin–orbit transduction (the coupling of an electron’s angular momentum with its linear momentum) combined with magnetoelectric switching. The device uses advanced quantum materials, especially correlated oxides and topological states of matter, for collective switching and detection. We describe progress in magnetoelectric switching and spin–orbit detection of state, and show that in comparison with CMOS technology our device has superior switching energy (by a factor of 10 to 30), lower switching voltage (by a factor of 5) and enhanced logic density (by a factor of 5). In addition, its non-volatility enables ultralow standby power, which is critical to modern computing. The properties of our device indicate that the proposed technology could enable the development of multi-generational computing. A scalable spintronic device operating via spin–orbit transduction and magnetoelectric switching and using advanced quantum materials shows non-volatility and improved performance and energy efficiency compared with CMOS devices.

Scalable energy-efficient magnetoelectric spin-orbit logic.

Methods based on deep neural networks with a massive number of layers and skip-connections have made impressive improvements on single image super-resolution (SISR). The skip-connections in these complex models boost the performance at the cost of a large amount of memory. With the increase of camera resolution from 1 million pixels to 100 million pixels on mobile phones, the memory footprint of these algorithms also increases hundreds of times, which restricts the applicability of these models on memory-limited devices. A plain model consisting of a stack of 3×3 convolutions with ReLU, in contrast, has the highest memory efficiency but poorly performs on super-resolution. This paper aims at calculating a winning initialization from a complex teacher network for a plain student network, which can provide performance comparable to complex models. To this end, we convert the teacher model to an equivalent large plain model and derive the plain student’s initialization. We further improve the student’s performance through initialization-aware feature distillation. Extensive experiments suggest that the proposed method results in a model with a competitive trade-off between accuracy and speed at a much lower memory footprint than other state-of-the-art lightweight approaches.

MemSR: Training Memory-efficient Lightweight Model for Image Super-Resolution

Single image dehazing is a challenging problem that aims to recover a high-quality haze-free image from a hazy image. In this paper, we propose an U-Net like deep network with contiguous memory residual blocks and gated fusion sub-network module to deal with the single image dehazing problem. The contiguous memory residual block is used to increase the flow of information by feature reusing and a gated fusion sub-network module is used to better combine the features of different levels. We evaluate our proposed method using two public image dehazing benchmarks. The experiments demonstrate that our network can achieve a state-of-the-art performance when compared with other popular methods.

Gated Contiguous Memory U-Net for Single Image Dehazing

Differential Power Analysis (DPA) reveals the secret key from the cryptographic device by side channel power leakage Masking and Random Delay Insertion (RDI) are two effective countermeasures against DPA attack This paper propose a novel countermeasure which associating masking with RDI, further, Multi-Masking instead of Transformed Masking is proposed in order to defend DPA attack based on hamming distance model The combined countermeasure is implemented on Data Encryption Standard (DES) The results show that combined countermeasure defends DPA attack with 105 power traces, and increases 40% ability against DPA

A combined countermeasure against DPA and implementation on DES

Many existing interpretation methods of convolutional neural networks (CNNs) mainly analyze in spatial domain, yet model interpretability in frequency domain has been rarely studied. To the best of our knowledge, there is no study on the interpretation of modern CNNs from the perspective of the frequency proportion of filters. In this work, we analyze the frequency properties of filters in the first layer as it is the entrance of information and relatively more convenient for analysis. By controlling the proportion of different frequency filters in the training stage, the network classification accuracy and model robustness is evaluated and our results reveal that it has a great impact on the robustness to common corruptions. Moreover, a learnable modulation of frequency proportion with perturbation in power spectrum is proposed from the perspective of frequency domain. Experiments on CIFAR-10-C show 10.97% average robustness gains for ResNet-18 with negligible natural accuracy degradation.

Exploring Frequency Domain Interpretation of Convolutional Neural Networks

It is well acknowledged in image processing domain that the information can be decomposed into different frequency parts and each part has its own merits. However, existing neural networks always ignore the distinctions and straightforwardly feed all the information into neural networks together, treating them equally. In this paper, we propose a novel neural networks framework named J-Net that decomposes images into different frequency bands and then processes them sequentially. Concretely, the images have been decomposed by wavelet transformation and then the wavelet coefficients are fed into neural networks gradually in different depth according to their decomposition levels. An attention module is utilized to facilitate the fusion of neural network features and injected information, yielding significant performance gain. Furthermore, we show how does the information with different frequency impact the accuracy of neural networks. Experiments show that 5.91%, 5.32% and 2.00% accuracy improvements on Caltech 101, Caltech256 and ImageNet, respectively.

Kaisheng Ma

Papers

MemSR: Training Memory-efficient Lightweight Model for Image Super-Resolution

Gated Contiguous Memory U-Net for Single Image Dehazing

A combined countermeasure against DPA and implementation on DES

Exploring Frequency Domain Interpretation of Convolutional Neural Networks

Wavelet J-Net: A Frequency Perspective on Convolutional Neural Networks