Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Brains, it has recently been argued, are essentially prediction machines. They are bundles of cells that support perception and action by constantly attempting to match incoming sensory inputs with top-down expectations or predictions. This is achieved using a hierarchical generative model that aims to minimize prediction error within a bidirectional cascade of cortical processing. Such accounts offer a unifying model of perception and action, illuminate the functional role of attention, and may neatly capture the special contribution of cortical processing to adaptive success. This target article critically examines this "hierarchical prediction machine" approach, concluding that it offers the best clue yet to the shape of a unified science of mind and action. Sections 1 and 2 lay out the key elements and implications of the approach. Section 3 explores a variety of pitfalls and challenges, spanning the evidential, the methodological, and the more properly conceptual. The paper ends (sections 4 and 5) by asking how such approaches might impact our more general vision of mind, experience, and agency.

/pdf/whatever-next-predictive-brains-situated-agents-and-the-epj56cs868.pdf

Whatever next? Predictive brains, situated agents, and the future of cognitive science

A central challenge for cognitive and systems neuroscience is to relate the incredibly complex behavior of animals to the equally complex activity of their brains. Recently described, large-scale neural models have not bridged this gap between neural activity and biological function. In this work, we present a 2.5-million-neuron model of the brain (called “Spaun”) that bridges this gap by exhibiting many different behaviors. The model is presented only with visual image sequences, and it draws all of its responses with a physically modeled arm. Although simplified, the model captures many aspects of neuroanatomy, neurophysiology, and psychological behavior, which we demonstrate via eight diverse tasks.

http://science.sciencemag.org/content/sci/338/6111/1202.full.pdf

A Large-Scale Model of the Functioning Brain

For years, researchers have used the theoretical tools of engineering to understand neural systems, but much of this work has been conducted in relative isolation. In Neural Engineering, Chris Eliasmith and Charles Anderson provide a synthesis of the disparate approaches current in computational neuroscience, incorporating ideas from neural coding, neural computation, physiology, communications theory, control theory, dynamics, and probability theory. This synthesis, they argue, enables novel theoretical and practical insights into the functioning of neural systems. Such insights are pertinent to experimental and computational neuroscientists and to engineers, physicists, and computer scientists interested in how their quantitative tools relate to the brain. The authors present three principles of neural engineering based on the representation of signals by neural ensembles, transformations of these representations through neuronal coupling weights, and the integration of control theory and neural dynamics. Through detailed examples and in-depth discussion, they make the case that these guiding principles constitute a useful theory for generating large-scale models of neurobiological function. A software package written in MatLab for use with their methodology, as well as examples, course notes, exercises, documentation, and other material, are available on the Web.

Neural engineering : computation, representation, and dynamics in neurobiological systems

Sequential model-based optimization (also known as Bayesian optimization) is one of the most efficient methods (per function evaluation) of function minimization. This efficiency makes it appropriate for optimizing the hyperparameters of machine learning algorithms that are slow to train. The Hyperopt library provides algorithms and parallelization infrastructure for performing hyperparameter optimization (model selection) in Python. This paper presents an introductory tutorial on the usage of the Hyperopt library, including the description of search spaces, minimization (in serial and parallel), and the analysis of the results collected in the course of minimization. This paper also gives an overview of Hyperopt-Sklearn, a software project that provides automatic algorithm configuration of the Scikit-learn machine learning library. Following Auto-Weka, we take the view that the choice of classifier and even the choice of preprocessing module can be taken together to represent a single large hyperparameter optimization problem. We use Hyperopt to define a search space that encompasses many standard components (e.g. SVM, RF, KNN, PCA, TFIDF) and common patterns of composing them together. We demonstrate, using search algorithms in Hyperopt and standard benchmarking data sets (MNIST, 20-newsgroups, convex shapes), that searching this space is practical and effective. In particular, we improve on best-known scores for the model space for both MNIST and convex shapes. The paper closes with some discussion of ongoing and future work.

https://iopscience.iop.org/article/10.1088/1749-4699/8/1/014008/pdf

Hyperopt: a Python library for model selection and hyperparameter optimization

Neuroscience currently lacks a comprehensive theory of how cognitive processes can be implemented in a biological substrate. The Neural Engineering Framework (NEF) proposes one such theory, but has not yet gathered significant empirical support, partly due to the technical challenge of building and simulating large-scale models with the NEF. Nengo is a software tool that can be used to build and simulate large-scale models based on the NEF; currently, it is the primary resource for both teaching how the NEF is used, and for doing research that generates specific NEF models to explain experimental data. Nengo 1.4, which was implemented in Java, was used to create Spaun, the world’s largest functional brain model (Eliasmith et al., 2012). Simulating Spaun highlighted limitations in Nengo 1.4’s ability to support model construction with simple syntax, to simulate large models quickly, and to collect large amounts of data for subsequent analysis. This paper describes Nengo 2.0, which is implemented in Python and overcomes these limitations. It uses simple and extendable syntax, simulates a benchmark model on the scale of Spaun 50 times faster than Nengo 1.4, and has a flexible mechanism for collecting simulation results.

/pdf/nengo-a-python-tool-for-building-large-scale-functional-3cp563g6lc.pdf

Nengo: a Python tool for building large-scale functional brain models.

Contents 1 The science of cognition 1.1 The last 50 years 1.2 How we got here 1.3 Where we are 1.4 Questions and answers 1.5 Nengo: An introduction Part I: How to build a brain 2 An introduction to brain building 2.1 Brain parts 2.2 A framework for building a brain 2.2.1 Representation 2.2.2 Transformation 2.2.3 Dynamics 2.2.4 The three principles 2.3 Levels 2.4 Nengo: Neural representation 3 Biological cognition - Semantics 3.1 The semantic pointer hypothesis 3.2 What is a semantic pointer? 3.3 Semantics: An overview 3.4 Shallow semantics 3.5 Deep semantics for perception 3.6 Deep semantics for action 3.7 The semantics of perception and action 3.8 Nengo: Neural computations 4 Biological cognition - Syntax 4.1 Structured representations 4.2 Binding without neurons 4.3 Binding with neurons. 4.4 Manipulating structured representations 4.5 Learning structural manipulations 4.6 Clean-up memory and scaling 4.7 Example: Fluid intelligence 4.8 Deep semantics for cognition 4.9 Nengo: Structured representations in neurons 5 Biological cognition - Control 5.1 The flow of information 5.2 The basal ganglia 5.3 Basal ganglia, cortex, and thalamus 5.4 Example: Fixed sequences of actions 5.5 Attention and the routing of information 5.6 Example: Flexible sequences of actions 5.7 Timing and control 5.8 Example: The Tower of Hanoi 5.9 Nengo: Question answering 6 Biological cognition - Memory and learning 6.1 Extending cognition through time 6.2 Working memory 6.3 Example: Serial list memory 6.4 Biological learning 6.5 Example: Learning new actions 6.6 Example: Learning new syntactic manipulations 6.7 Nengo: Learning 7 The Semantic Pointer Architecture (SPA) 7.1 A summary of the SPA 7.2 A SPA unified network 7.3 Tasks 7.3.1 Recognition 7.3.2 Copy drawing 7.3.3 Reinforcement learning 7.3.4 Serial working memory 7.3.5 Counting 7.3.6 Question answering 7.3.7 Rapid variable creation 7.3.8 Fluid reasoning 7.3.9 Discussion 7.4 A unified view: Symbols and probabilities 7.5 Nengo: Advanced modeling methods Part II Is that how you build a brain? 8 Evaluating cognitive theories 341 8.1 Introduction 8.2 Core cognitive criteria (CCC) 8.2.1 Representational structure 8.2.1.1 Systematicity 8.2.1.2 Compositionality 8.2.1.3 Productivity 8.2.1.4 The massive binding problem 8.2.2 Performance concerns 8.2.2.1 Syntactic generalization 8.2.2.2 Robustness 8.2.2.3 Adaptability 8.2.2.4 Memory 8.2.2.5 Scalability 8.2.3 Scientific merit 8.2.3.1 Triangulation (contact with more sources of data) 8.2.3.2 Compactness 8.3 Conclusion 8.4 Nengo Bonus: How to build a brain - a practical guide 9 Theories of cognition 9.1 The state of the art 9.1.1 ACT-R 9.1.2 Synchrony-based approaches 9.1.3 Neural blackboard architecture (NBA) 9.1.4 The integrated connectionist/symbolic architecture (ICS) 9.1.5 Leabra 9.1.6 Dynamic field theory (DFT) 9.2 An evaluation 9.2.1 Representational structure 9.2.2 Performance concerns 9.2.3 Scientific merit 9.2.4 Summary 9.3 The same... 9.4 ...but different 9.5 The SPA versus the SOA 10 Consequences and challenges 10.1 Representation 10.2 Concepts 10.3 Inference 10.4 Dynamics 10.5 Challenges 10.6 Conclusion A Mathematical notation and overview A.1 Vectors A.2 Vector spaces A.3 The dot product A.4 Basis of a vector space A.5 Linear transformations on vectors A.6 Time derivatives for dynamics B Mathematical derivations for the NEF B.1 Representation B.1.1 Encoding B.1.2 Decoding B.2 Transformation B.3 Dynamics C Further details on deep semantic models C.1 The perceptual model C.2 The motor model D Mathematical derivations for the SPA D.1 Binding and unbinding HRRs D.2 Learning high-level transformations D.3 Ordinal serial encoding model D.4 Spike-timing dependent plasticity D.5 Number of neurons for representing structure E SPA model details E.1 Tower of Hanoi Bibliography Index

Chris Eliasmith

Papers

A Large-Scale Model of the Functioning Brain

Neural engineering : computation, representation, and dynamics in neurobiological systems

Hyperopt: a Python library for model selection and hyperparameter optimization

Nengo: a Python tool for building large-scale functional brain models.

How to Build a Brain: A Neural Architecture for Biological Cognition