scispace - formally typeset
Search or ask a question
Journal ArticleDOI

Blind separation of sources, Part 1: an adaptive algorithm based on neuromimetic architecture

01 Aug 1991-Signal Processing (Elsevier North-Holland, Inc.)-Vol. 24, Iss: 1, pp 1-10
TL;DR: A new concept, that of INdependent Components Analysis (INCA), more powerful than the classical Principal components Analysis (in decision tasks) emerges from this work.
About: This article is published in Signal Processing.The article was published on 1991-08-01. It has received 2583 citations till now. The article focuses on the topics: Blind signal separation & Adaptive algorithm.
Citations
More filters
Book
18 Nov 2016
TL;DR: Deep learning as mentioned in this paper is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts, and it is used in many applications such as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames.
Abstract: Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

38,208 citations

Journal ArticleDOI
TL;DR: This historical survey compactly summarizes relevant work, much of it from the previous millennium, review deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

14,635 citations


Cites background from "Blind separation of sources, Part 1..."

  • ...d Hinton, 1994; Field, 1994; Hinton et al., 1995; Dayan and Zemel, 1995; Amari et al., 1996; Deco and Parra, 1997). Many do this to uncover and disentangle hidden underlying sources of signals (e.g., Jutten and Herault, 1991; Schuster, 1992; Andrade et al., 1993; Molgedey and Schuster, 1994; Comon, 1994; Cardoso, 1994; Bell and Sejnowski, 1995; Karhunenand Joutsensalo, 1995; Belouchrani et al., 1997; Hyva¨rinen et al., 2...

    [...]

  • ...…1993; Bell & Sejnowski, 1995; Belouchrani, Abed-Meraim, Cardoso, & Moulines, 1997; Cardoso, 1994; Comon, 1994; Hyvärinen, Karhunen, & Oja, 2001; Jutten & Herault, 1991; Karhunen & Joutsensalo, 1995; Molgedey & Schuster, 1994; Schuster, 1992; Shan & Cottrell, 2014; Shan, Zhang, & Cottrell,…...

    [...]

Journal ArticleDOI
21 Oct 1999-Nature
TL;DR: An algorithm for non-negative matrix factorization is demonstrated that is able to learn parts of faces and semantic features of text and is in contrast to other methods that learn holistic, not parts-based, representations.
Abstract: Is perception of the whole based on perception of its parts? There is psychological and physiological evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.

11,500 citations

Journal ArticleDOI
TL;DR: Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks.
Abstract: The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

11,201 citations


Cites background from "Blind separation of sources, Part 1..."

  • ...Similarly, Le et al. (2011b) recently showed that adding a regularization term of the form ∑ t ∑ j s3(Wjx (t)) to a linear auto-encoder with tied weights, where s3 is a nonlinear convex function, yields an efficient algorithm for learning linear ICA....

    [...]

  • ...t al., 2011c; Chen et al., 2012). Another rich family of feature extraction techniques that this review does not cover in any detail due to space constraints is Independent Component Analysis or ICA (Jutten and Herault, 1991; Comon, 1994; Bell and Sejnowski, 1997). Instead, we refer the reader to Hyvarinen¨ et al. (2001a); Hyvarinen¨ et al. (2009). Note that, while in the simplest case (complete, noisefree) ICA yields li...

    [...]

  • ...Therefore, ICA and its variants like Independent and Topographic ICA (Hyvärinen et al., 2001b) can and have been used to build deep networks (Le et al., 2010, 2011c): see section 11.2....

    [...]

  • ...Another rich family of feature extraction techniques that this review does not cover in any detail due to space constraints is Independent Component Analysis or ICA (Jutten and Herault, 1991; Bell and Sejnowski, 1997)....

    [...]

  • ...Note that, while in the simplest case (complete, noise-free) ICA yields linear features, in the more general case it can be equated with a linear generative model with non-Gaussian independent latent variables, similar to sparse coding (section 6.1.1), which result in non-linear features....

    [...]

Christopher M. Bishop1
01 Jan 2006
TL;DR: Probability distributions of linear models for regression and classification are given in this article, along with a discussion of combining models and combining models in the context of machine learning and classification.
Abstract: Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

10,141 citations

References
More filters
Book
01 Jan 1989
TL;DR: This chapter discusses a simple circuit that can generate a sinusoidal response and calls this circuit the second-order section, which can be used to generate any response that can be represented by two poles in the complex plane, where the two poles have both real and imaginary parts.

2,460 citations

Journal ArticleDOI
TL;DR: An historical discussion is provided of the intellectual trends that caused nineteenth century interdisciplinary studies of physics and psychobiology by leading scientists such as Helmholtz, Maxwell, and Mach to splinter into separate twentieth-century scientific movements.

1,586 citations

Journal ArticleDOI
TL;DR: It was confirmed that most cortical neurones prefer vertical stimulus orientations when experience is restricted to vertical contours in both eyes, and that, if the experienced orientations are different in the two eyes, each eye dominates over those neurones whose orientation preference corresponds to the orientation this eye has experienced.
Abstract: 1. Kittens were dark-reared until 4-6 weeks old, and then for another 4-7 weeks with various combinations of cylindrical lenses, monocular occlusion, and normal vision. 2. Single unit recordings from 816 neurones of the visual cortex (area 17) were obtained after the end of exposure. Clear-cut effects on the distributions of the neurones' ocular dominance and orientation preference were found yielding close correlations with the rearing conditions. 3. It was confirmed that most cortical neurones prefer vertical stimulus orientations when experience is restricted to vertical contours in both eyes. It was further confirmed that, if the experienced orientations are different in the two eyes, each eye dominates over those neurones whose orientation preference corresponds to the orientation this eye has experienced. 4. When one eye is covered while the other sees only contours of one orientation, the ocular dominance distribution of cortical neurones shows a bias towards the open eye. Neurones dominated by this eye prefer orientations corresponding to the experienced range. Neurones preferring other orientations are shared between both eyes. 5. When vision is unimpaired in one eye and restricted to vertical contours in the other, binocularity is common among neurones preferring vertical orientations. Neurones with orientation preferences off the vertical are mainly monocular and dominated by the eye with unrestricted vision. 6. When normal monocular vision of one eye precedes restricted monocular vision of the other eye, only a few binocular units are encountered. Reversal of the initial effects of monocular experience is found only in neurones preferring the orientation that has been experienced by the newly opened eye. The other neurones remain dominated by the originally open eye. Thus, complementary distributions of orientation preferences are found for the two eyes. 7. A good correlation was found between the amount of orientational experience as determined by the number of orientations exposed and the number of normally tuned neurones. Conversely, the number of neurones responding to all orientations decreases with increasing amount of experience.

219 citations


"Blind separation of sources, Part 1..." refers background in this paper

  • ...Relations (24) and (25) are very close to mathematic formalizations [6] of rules for synaptic plasticity proposed as by Hebb [7] or Rauschecker and Singer [15]....

    [...]

Dissertation
01 Jan 1987
TL;DR: Analyse rigoureuse sur le traitement de signaux multidimensionnels par analogie au systeme nerveux central is presented in this article, which is based on the analysis of signaux multi-dimensionnels.
Abstract: Analyse rigoureuse sur le traitement de signaux multidimensionnels par analogie au systeme nerveux central

54 citations