We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

Adam: A Method for Stochastic Optimization

Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1–4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence—the structure prediction component of the ‘protein folding problem’8—has been an important open research problem for more than 50 years9. Despite recent progress10–14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm. AlphaFold predicts protein structures with an accuracy competitive with experimental structures in the majority of cases using a novel deep learning architecture.

/pdf/highly-accurate-protein-structure-prediction-with-alphafold-2xjbzfwu3n.pdf

Highly accurate protein structure prediction with AlphaFold

https://research.cs.queensu.ca/home/shatkay/879papers/TargetPEmanuelsson.pdf

Predicting subcellular localization of proteins based on their N-terminal amino acid sequence.

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

An overview of gradient descent optimization algorithms

Several means for improving the performance and training of neural networks for classification are proposed Crossvalidation is used as a tool for optimizing network parameters and architecture It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks >

/pdf/neural-network-ensembles-4ujxrg94mn.pdf

Neural network ensembles

On the momentum term in gradient descent learning algorithms

http://papers.cnl.salk.edu/PDFs/Predicting%20the%20Secondary%20Structure%20of%20Globular%20Proteins%20Using%20Neural%20Network%20Models%201988-3749.pdf

Predicting the secondary structure of globular proteins using neural network models.

The adult brain shows remarkable plasticity, as demonstrated by the improvement in fine sensorial discriminations after intensive practice. The behavioural aspects of such perceptual learning are well documented, especially in the visual system1,2,3,4,5,6,7,8. Specificity for stimulus attributes clearly implicates an early cortical site, where receptive fields retain fine selectivity for these attributes; however, the neuronal correlates of a simple visual discrimination task remained unidentified. Here we report electrophysiological correlates in the primary visual cortex (V1) of monkeys for learning orientation identification. We link the behavioural improvement in this type of learning to an improved neuronal performance of trained compared to naive neurons. Improved long-term neuronal performance resulted from changes in the characteristics of orientation tuning of individual neurons. More particularly, the slope of the orientation tuning curve that was measured at the trained orientation increased only for the subgroup of trained neurons most likely to code the orientation identified by the monkey. No modifications of the tuning curve were observed for orientations for which the monkey had not been trained. Thus training induces a specific and efficient increase in neuronal sensitivity in V1.

/pdf/practising-orientation-identification-improves-orientation-5alrcaj0bq.pdf

Practising orientation identification improves orientation coding in V1 neurons

We investigated how the primate visual system solves the difficult problem of representing multiple motion vectors in the same part of the visual space--the problem of motion transparency. In the preceding companion article we reported that displays with locally well-balanced motion signals in opposite directions are perceptually nontransparent (i.e., one does not see two coherent moving surfaces) and that transparent displays always contain locally unbalanced motion signals. This is exemplified by our paired and unpaired dot patterns. Although both types of stimuli contain two sets of dots moving in opposite directions, the former is locally well balanced and appears like flicker while the latter gives a perception of two transparent surfaces. In this article we report our physiological recordings from areas V1 and MT of behaving monkeys, comparing single-cell responses to the paired and the unpaired dot patterns. Although a small proportion of directionally selective V1 cells responded differently to the two types of patterns, the average V1 responses could not reliably distinguish between the paired and the unpaired stimuli. A large fraction of MT cells, on the other hand, responded significantly better to the unpaired dot patterns than to the paired ones. Furthermore, the average response of all MT cells to the unpaired dot patterns was significantly higher than that to the paired dot patterns. These results demonstrate a neural correlate of the perceptual transparency at the level of MT. On the other hand, V1 cells do not generally discriminate between the transparent and nontransparent stimuli, indicating that V1 activity is not well correlated with the perception of motion transparency. Our results are consistent with a two-stage model for motion processing: the first stage measures local motion and the second stage introduces suppression if different directions of motion are present at a local region of the visual field. The first stage is located primarily in V1 and the second stage primarily in MT. Finally, we found a strong and negative correlation between the degree of the opponent-direction suppression of MT cells and their responses to flicker noise stimuli. This result suggests that one of the fundamental roles of the opponent-direction suppression in MT is noise reduction.

https://www.jneurosci.org/content/jneuro/14/12/7367.full.pdf

Transparent motion perception as detection of unbalanced motion signals. II. Physiology

Our visual system can solve the difficult problem of representing multiple motions in the same part of the visual space, the motion transparency problem. We investigated the conditions under which transparent motion perception occurs through psychophysical observations, using a series of visual displays composed of two simple patterns moving in opposite directions. We found that whenever a display has finely balanced opposing motion signals in all local regions, it is perceptually nontransparent. The displays that appeared transparent always contain locally unbalanced motion signals, with some local regions having net motion signals in one direction and some other regions in the opposite direction. These interdigitating net motion signals in both directions appear to be integrated separately to form two overlapping transparent surfaces. Displays that were spatially balanced could be made perceptually transparent if the two components moving in opposite directions were at different stereo depth planes or had different spatial frequency contents. Our results can be explained by proposing a disparity- and spatial frequency-specific suppression stage in the motion pathway, at which motion signals of different directions, but of the same disparity and spatial frequency contents, locally inhibit each other. Such a mechanism would suppress noise input to the motion system, which generally activates several direction channels simultaneously, and would still not eliminate activity evoked by transparent surfaces that are at different depths or have different textures.

http://persci.mit.edu/pub_pdfs/qian94_1.pdf

Ning Qian

Papers

On the momentum term in gradient descent learning algorithms

Predicting the secondary structure of globular proteins using neural network models.

Practising orientation identification improves orientation coding in V1 neurons

Transparent motion perception as detection of unbalanced motion signals. II. Physiology

Transparent Motion Perception as Detection of Unbalanced Motion Signals. I. Psychophysics