5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Deep learning is at the heart of the current rise of artificial intelligence. In the field of computer vision, it has become the workhorse for applications ranging from self-driving cars to surveillance and security. Whereas, deep neural networks have demonstrated phenomenal success (often beyond human capabilities) in solving complex problems, recent studies show that they are vulnerable to adversarial attacks in the form of subtle perturbations to inputs that lead a model to predict incorrect outputs. For images, such perturbations are often too small to be perceptible, yet they completely fool the deep learning models. Adversarial attacks pose a serious threat to the success of deep learning in practice. This fact has recently led to a large influx of contributions in this direction. This paper presents the first comprehensive survey on adversarial attacks on deep learning in computer vision. We review the works that design adversarial attacks, analyze the existence of such attacks and propose defenses against them. To emphasize that adversarial attacks are possible in practical conditions, we separately review the contributions that evaluate adversarial attacks in the real-world scenarios. Finally, drawing on the reviewed literature, we provide a broader outlook of this research direction.

Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey

We review the problem of dark energy, including a survey of theoretical models and some aspects of numerical studies

Dark Energy

This article is devoted to the perturbative renormalization of the abelian Higgs-Kibble model, within the class of renormalizable gauges which are odd under charge conjugation. The Bogoliubov Parasiuk Hepp-Zimmermann renormalization scheme is used throughout, including the renormalized action principle proved by Lowenstein and Lam. The whole study is based on the fulfillment to all orders of perturbation theory of the Slavnov identities which express the invariance of the Lagrangian under a supergauge type family of non-linear transformations involving the Faddeev-Popov ghosts. Direct combinatorial proofs are given of the gauge independence and unitarity of the physicalS operator. Their simplicity relies both on a systematic use of the Slavnov identities as well as suitable normalization conditions which allow to perform all mass renormalizations, including those pertaining to the ghosts, so that the theory can be given a setting within a fixed Fock space. Some simple gauge independent local operators are constructed.

/pdf/renormalization-of-the-abelian-higgs-kibble-model-14fr0vp7oj.pdf

Renormalization of the Abelian Higgs-Kibble Model

Machine learning models are known to lack robustness against inputs crafted by an adversary. Such adversarial examples can, for instance, be derived from regular inputs by introducing minor—yet carefully selected—perturbations.

Adversarial examples for malware detection

Infrared Sensitivity of Unstable Vacua

It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb's idea that change of the synapse strength should be local-i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.

https://www.pnas.org/content/pnas/116/16/7723.full.pdf

Unsupervised learning by competing hidden units

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.

Dense Associative Memory for Pattern Recognition

Spatial patterns in the early fruit fly embryo emerge from a network of interactions among transcription factors, the gap genes, driven by maternal inputs. Such networks can exhibit many qualitatively different behaviors, separated by critical surfaces. At criticality, we should observe strong correlations in the fluctuations of different genes around their mean expression levels, a slowing of the dynamics along some but not all directions in the space of possible expression levels, correlations of expression fluctuations over long distances in the embryo, and departures from a Gaussian distribution of these fluctuations. Analysis of recent experiments on the gap gene network shows that all these signatures are observed, and that the different signatures are related in ways predicted by theory. Although there might be other explanations for these individual phenomena, the confluence of evidence suggests that this genetic network is tuned to criticality.

https://www.pnas.org/content/pnas/111/10/3683.full.pdf

Morphogenesis at criticality

/pdf/dense-associative-memory-for-pattern-recognition-3mtdhyh0uv.pdf

Dmitry Krotov

Papers

Infrared Sensitivity of Unstable Vacua

Unsupervised learning by competing hidden units

Dense Associative Memory for Pattern Recognition

Morphogenesis at criticality

Dense Associative Memory for Pattern Recognition