A New Learning Algorithm for Blind Signal Separation

Home
/
Papers
/
A New Learning Algorithm for Blind Signal Separation

Proceedings Article•

A New Learning Algorithm for Blind Signal Separation

Shun-ichi Amari¹, Andrzej Cichocki, Howard Hua Yang•Institutions (1)

27 Nov 1995-Vol. 8, pp 757-763

TL;DR: A new on-line learning algorithm which minimizes a statistical dependency among outputs is derived for blind separation of mixed signals and has an equivariant property and is easily implemented on a neural network like model.

read less

Abstract: A new on-line learning algorithm which minimizes a statistical dependency among outputs is derived for blind separation of mixed signals. The dependency is measured by the average mutual information (MI) of the outputs. The source signals and the mixing matrix are unknown except for the number of the sources. The Gram-Charlier expansion instead of the Edgeworth expansion is used in evaluating the MI. The natural gradient approach is used to minimize the MI. A novel activation function is proposed for the on-line learning algorithm which has an equivariant property and is easily implemented on a neural network like model. The validity of the new learning algorithm are verified by computer simulations.

...read moreread less

Content maybe subject to copyright Report

Citations

PDF

Open Access

More filters

Journal Article•DOI•

EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.

[...]

Arnaud Delorme¹, Scott Makeig¹•Institutions (1)

University of California, San Diego¹

15 Mar 2004-Journal of Neuroscience Methods

TL;DR: EELAB as mentioned in this paper is a toolbox and graphic user interface for processing collections of single-trial and/or averaged EEG data of any number of channels, including EEG data, channel and event information importing, data visualization (scrolling, scalp map and dipole model plotting, plus multi-trial ERP-image plots), preprocessing (including artifact rejection, filtering, epoch selection, and averaging), Independent Component Analysis (ICA) and time/frequency decomposition including channel and component cross-coherence supported by bootstrap statistical methods based on data resampling.

...read moreread less

17,362 citations

Journal Article•DOI•

Deep learning in neural networks

[...]

Jürgen Schmidhuber¹•Institutions (1)

University of Lugano¹

01 Jan 2015-Neural Networks

TL;DR: This historical survey compactly summarizes relevant work, much of it from the previous millennium, review deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

...read moreread less

14,635 citations

Additional excerpts

...Many UL methods are designed to maximize information-theoretic objectives (e.g., Linsker, 1988; Barlow et al., 1989; MacKay and Miller, 1990; Plumbley, 1991; Schmidhuber, 1992b,c; Schraudolph and Sejnowski, 1993; Redlich, 1993; Zemel, 1993; Zemel and Hinton, 1994; Field, 1994; Hinton et al., 1995; Dayan and Zemel, 1995; Amari et al., 1996; Deco and Parra, 1997), and to uncover and disentangle hidden underlying...
[...]

Pattern Recognition and Machine Learning

[...]

Christopher M. Bishop¹•Institutions (1)

Microsoft¹

01 Jan 2006

TL;DR: Probability distributions of linear models for regression and classification are given in this article, along with a discussion of combining models and combining models in the context of machine learning and classification.

...read moreread less

Abstract: Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

...read moreread less

10,141 citations

Journal Article•DOI•

Independent component analysis: algorithms and applications

[...]

Aapo Hyvärinen¹, Erkki Oja¹•Institutions (1)

Helsinki University of Technology¹

01 May 2000-Neural Networks

TL;DR: The basic theory and applications of ICA are presented, and the goal is to find a linear representation of non-Gaussian data so that the components are statistically independent, or as independent as possible.

...read moreread less

8,231 citations

Cites methods from "A New Learning Algorithm for Blind ..."

...The above version of FastICA could be compared with the stochastic gradient method for maximizing likelihood ( Amari et al., 1996; Bell and Sejnowski, 1995; Cardoso and Laheld, 1996; Cichocki and Unbehauen, 1996):...
[...]
...Finally, we give a version of FastICA that shows explicitly the connection to the well-known infomax or maximum likelihood algorithm introduced in ( Amari et al., 1996; Bell and Sejnowski, 1995; Cardoso and Laheld, 1996; Cichocki and Unbehauen, 1996)....
[...]

Book•

Information Theory, Inference and Learning Algorithms

[...]

David J. C. MacKay

06 Oct 2003

TL;DR: A fun and exciting textbook on the mathematics underpinning the most dynamic areas of modern science and engineering.

...read moreread less

Abstract: Fun and exciting textbook on the mathematics underpinning the most dynamic areas of modern science and engineering.

...read moreread less

8,091 citations

Additional excerpts

...Further reading on blind separation, including non-ICA algorithms, can be found in (Jutten and Herault, 1991; Comon et al., 1991; Hendin et al., 1994; Amari et al., 1996; Hojen-Sorensen et al., 2002)....
[...]

1
2
3
4
…
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

Collapse

References

PDF

Open Access

More filters

Journal Article•DOI•

An information-maximization approach to blind separation and blind deconvolution

[...]

Anthony J. Bell¹, Terrence J. Sejnowski¹•Institutions (1)

University of California, San Diego¹

01 Nov 1995-Neural Computation

TL;DR: It is suggested that information maximization provides a unifying framework for problems in "blind" signal processing and dependencies of information transfer on time delays are derived.

...read moreread less

Abstract: We derive a new self-organizing learning algorithm that maximizes the information transferred in a network of nonlinear units. The algorithm does not assume any knowledge of the input distributions, and is defined here for the zero-noise limit. Under these conditions, information maximization has extra properties not found in the linear case (Linsker 1989). The nonlinearities in the transfer function are able to pick up higher-order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation. This enables the network to separate statistically independent components in the inputs: a higher-order generalization of principal components analysis. We apply the network to the source separation (or cocktail party) problem, successfully separating unknown mixtures of up to 10 speakers. We also show that a variant on the network architecture is able to perform blind deconvolution (cancellation of unknown echoes and reverberation in a speech signal). Finally, we derive dependencies of information transfer on time delays. We suggest that information maximization provides a unifying framework for problems in "blind" signal processing.

...read moreread less

9,157 citations

"A New Learning Algorithm for Blind ..." refers background or methods in this paper

...Although the on-line learning algorithms (16) and (19) look similar to those in [3, 7] and [5] respectively, the selection of the activation function in this paper is rational, not ad hoc....
[...]
...Several neural network algorithms [3, 5, 7] have been proposed for solving this problem....
[...]
...It is a non-monotonic activation function different from those used in [3, 5, 7]....
[...]

Journal Article•DOI•

Independent component analysis, a new concept?

[...]

Pierre Comon

01 Apr 1994-Signal Processing

TL;DR: An efficient algorithm is proposed, which allows the computation of the ICA of a data matrix within a polynomial time and may actually be seen as an extension of the principal component analysis (PCA).

...read moreread less

8,522 citations

"A New Learning Algorithm for Blind ..." refers background or methods in this paper

...The minimization of the Kullback-Leibler divergence leads to an ICA algorithm for estimating W in [6] where the Edgeworth expansion is used to evaluate the negentropy....
[...]
...In practice, other activation functions such as those proposed in [2]-[6] may also be used in (19)....
[...]
...The algorithm in [6] is based on the Edgeworth expansion[8] for evaluating the marginal negentropy....
[...]
...The mathematical framework for the ICA is formulated in [6]....
[...]
...Different from the work in [6], we use the Gram-Charlier expansion instead of the Edgeworth expansion to calculate the marginal entropy in evaluating the MI....
[...]

Journal Article•

Kendall's advanced theory of statistics

[...]

Maurice G. Kendall, Alan Stuart, J. K. Ord

07 Apr 2005

3,470 citations

Journal Article•DOI•

Blind separation of sources, Part 1: an adaptive algorithm based on neuromimetic architecture

[...]

Christian Jutten, Jeanny Hérault

01 Aug 1991-Signal Processing

TL;DR: A new concept, that of INdependent Components Analysis (INCA), more powerful than the classical Principal components Analysis (in decision tasks) emerges from this work.

...read moreread less

2,583 citations

"A New Learning Algorithm for Blind ..." refers background or methods in this paper

...Although the on-line learning algorithms (16) and (19) look similar to those in [3, 7] and [5] respectively, the selection of the activation function in this paper is rational, not ad hoc....
[...]
...Several neural network algorithms [3, 5, 7] have been proposed for solving this problem....
[...]
...How should the activation function be determined to minimize the MI? Is it necessary to use monotonic activation functions for blind signal separation? In this paper, we shall answer these questions and give an on-line learning algorithm which uses a non-monotonic activation function selected by the independent component analysis (ICA) [7]....
[...]
...It is a non-monotonic activation function different from those used in [3, 5, 7]....
[...]

Journal Article•DOI•

Equivariant adaptive source separation

[...]

Jean-François Cardoso¹, B.H. Laheld²•Institutions (2)

Télécom ParisTech¹, École Normale Supérieure²

01 Dec 1996-IEEE Transactions on Signal Processing

TL;DR: A class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called EASI, which yields algorithms with a simple structure for both real and complex mixtures.

...read moreread less

Abstract: Source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed. This paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence (EASI). The EASI algorithms are based on the idea of serial updating. This specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures. Most importantly, the performance of an EASI algorithm does not depend on the mixing matrix. In particular, convergence rates, stability conditions, and interference rejection levels depend only on the (normalized) distributions of the source signals. Closed-form expressions of these quantities are given via an asymptotic performance analysis. The theme of equivariance is stressed throughout the paper. The source separation problem has an underlying multiplicative structure. The parameter space forms a (matrix) multiplicative group. We explore the (favorable) consequences of this fact on implementation, performance, and optimization of EASI algorithms.

...read moreread less

1,417 citations