National Research University – Higher School of Economics

About: National Research University – Higher School of Economics is a education organization based out in Moscow, Russia. It is known for research contribution in the topics: Population & Computer science. The organization has 12873 authors who have published 23376 publications receiving 256396 citations.

Quantum continuous $\mathfrak{gl}_{\infty}$: Semiinfinite construction of representations

Abstract: We begin a study of the representation theory of quantum continuous gl∞, which we denote by E. This algebra depends on two parameters and is a deformed version of the enveloping algebra of the Lie algebra of difference operators acting on the space of Laurent polynomials in one variable. Fundamental representations of E are labeled by a continuous parameter u∈C. The representation theory of E has many properties familiar from the representation theory of gl∞: vector representations, Fock modules, and semiinfinite constructions of modules. Using tensor products of vector representations, we construct surjective homomorphisms from E to spherical double affine Hecke algebras SHN for all N. A key step in this construction is an identification of a natural basis of the tensor products of vector representations with Macdonald polynomials. We also show that one of the Fock representations is isomorphic to the module constructed earlier by means of the K-theory of Hilbert schemes.

Averaging Weights Leads to Wider Optima and Better Generalization

Abstract: Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

Structured Bayesian Pruning via Log-Normal Multiplicative Noise

Abstract: Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves gener- alization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural net- works and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is com- puted in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.

Heart–brain interactions shape somatosensory perception and evoked potentials

Abstract: Even though humans are mostly not aware of their heartbeats, several heartbeat-related effects have been reported to influence conscious perception. It is not clear whether these effects are distinct or related phenomena, or whether they are early sensory effects or late decisional processes. Combining electroencephalography and electrocardiography, along with signal detection theory analyses, we identify two distinct heartbeat-related influences on conscious perception differentially related to early vs. late somatosensory processing. First, an effect on early sensory processing was found for the heartbeat-evoked potential (HEP), a marker of cardiac interoception. The amplitude of the prestimulus HEP negatively correlated with localization and detection of somatosensory stimuli, reflecting a more conservative detection bias (criterion). Importantly, higher HEP amplitudes were followed by decreases in early (P50) as well as late (N140, P300) somatosensory-evoked potential (SEP) amplitudes. Second, stimulus timing along the cardiac cycle also affected perception. During systole, stimuli were detected and correctly localized less frequently, relating to a shift in perceptual sensitivity. This perceptual attenuation was accompanied by the suppression of only late SEP components (P300) and was stronger for individuals with a more stable heart rate. Both heart-related effects were independent of alpha oscillations' influence on somatosensory processing. We explain cardiac cycle timing effects in a predictive coding account and suggest that HEP-related effects might reflect spontaneous shifts between interoception and exteroception or modulations of general attentional resources. Thus, our results provide a general conceptual framework to explain how internal signals can be integrated into our conscious perception of the world.

Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data

Abstract: Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.