We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

[신간의 별자리x] 우리/미술, 그리고 ‘슬픔의 박물관’

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

/pdf/photo-realistic-single-image-super-resolution-using-a-2y0438rlmd.pdf

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Despite the remarkable performance on high-quality (HQ) data, the accuracy of deep image recognition models degrades rapidly in the presence of low-quality (LQ) images. Both feature de-drifting and quality agnostic models have been developed to make the features extracted from degraded images closer to those of HQ images. In these methods, the $$l_2$$ -norm is usually used as a constraint. It treats each pixel in the feature equally and may result in relatively poor reconstruction performance in some difficult regions. To address this issue, we propose a novel self-feature distillation method with uncertainty modeling for better producing HQ-like features from low-quality observations in this paper. Specifically, in a standard recognition model, we use the HQ features to distill the corresponding degraded ones and conduct uncertainty modeling according to the diversity of degradation sources to adaptively increase the weights of feature regions that are difficult to recover in the distillation loss. Experiments demonstrate that our method can extract HQ-like features better even when the inputs are degraded images, which makes the model more robust than other approaches. 

Self-feature Distillation with Uncertainty Modeling for Degraded Image Recognition

This paper presents a kind of basis functions used for image directional representation, which are obtained from the pseudo polar Fourier transform-based directional filter banks (PPFT-DFBs). The basis functions so obtained have the attractive properties of i) flexible direction and ii) arbitrary number of subbands. More specifically, we give the derivation of the basis function from the equivalent directional filter of PPFT-DFBs. Further, we analyze the properties of basis functions and their corresponding basis images. Experiments are given to testify the flexibility of basis functions and their efficiency in directional representation.

A flexible directional image representation using pseudo polar fourier transform based DFBs

Despite the remarkable performance on various Natural Language Processing (NLP) tasks, the parametric complexity of pretrained language models has remained a major obstacle due to limited computational resources in many practical applications. Techniques such as knowledge distillation, network pruning, and quantization have been developed for language model compression. However, it has remained challenging to achieve an optimal tradeoff between model size and inference accuracy. To address this issue, we propose a novel and efficient uncertainty-driven knowledge distillation compression method for transformer-based pretrained language models. Specifically, we design a method of parameter retention and feedforward network parameter distillation to compress N-stacked transformer modules into one module in the fine-tuning stage. A key innovation of our approach is to add the uncertainty estimation module (UEM) into the student network such that it can guide the student network's feature reconstruction in the latent space (similar to the teacher's). Across multiple datasets in the natural language inference tasks of GLUE, we have achieved more than 95% accuracy of the original BERT, while only using about 50% of the parameters. 

Uncertainty-Driven Knowledge Distillation for Language Model Compression

More and more complex Deep Neural Networks (DNNs) are designed for the improvement of human pose estimation task. However, it is still hard to handle the inherent ambiguities due to diversity of postures and occlusions. And it is difficult to meet the requirements for the high accuracy of human pose estimation in practical applications. In this paper, reasoning-based multi-level predictions with graphical model for single person human pose estimation is proposed to obtain the accurate location of body joints. Specifically, a multi-level prediction using cascaded network is designed with recursive prediction according to three different levels from easy to hard joints. At each stage, multi-scale fusion and channel-wise feature enhancement are employed for stronger contextual information to improve capacity of feature extraction. Heatmaps with rich spatial and semantic information are refined by explicitly constructing graphical model to learn the structure information for inference, which can implement the interactions between joints. The proposed method is evaluated on LSP dataset. The experiments show that it can achieve highly accurate results and outperform state-of-the-art methods.

Multi-level Prediction with Graphical Model for Human Pose Estimation.

Filter banks are found many applications for images and signal processing. A lower complex filter banks are desired for their effective implementation. In this paper, we address a problem how to design low complex filter banks. A perfect restructure (PR) filter banks can be factorized by lifting steps and the procedure of factorization are described as multi-fork tree-structure. A heuristic-searching algorithm was proposed for finding the optimization PR filter banks with low dynamical coefficients. The given examples show the proposed method is effective.

Guangming Shi

Papers

Self-feature Distillation with Uncertainty Modeling for Degraded Image Recognition

A flexible directional image representation using pseudo polar fourier transform based DFBs

Uncertainty-Driven Knowledge Distillation for Language Model Compression

Multi-level Prediction with Graphical Model for Human Pose Estimation.

Heuristic searching algorithm for design structurally perfect reconstruction low complex filter banks