998
IEEE
TRANSACTIONS
ON
NEURAL
NETWORKS,
VOL.
5,
NO.
6,
NOVEMBER
1994
Delay-Independent Stability in Bidirectional Associative Memory Networks
K.
Gopalsamy
and
Xue-Zhong
He
Abstract-It is shown that if the neuronal
gains
are small com-
pared with the synaptic connection weights, then
a
bidirectional
associative memory network with axonal signal transmission
delays converges to the equilibria associated with exogenous
inputs to the network; both discrete and continuously distributed
delays are considered; the asymptotic stability is global in the
state space of neuronal activations and also is independent of the
delays.
I.
INTRODUCTION
HE stability characteristics of equilibria of continuous
T
bidirectional associative memory networks of the type
n
i
=
1,2,...,n
;=1
--
dYi(t)
-
-Yi(t)
+
Cmijs(xj(t))
+
Ji
dt
j=1
and some of their generalizations have been investigated by
Kosko [9],
[lo].
Networks of the form
(1)
generalize the
continuous Hopfield circuit model
[8]
and can be obtained as a
special case from the model of Cohen and Grossberg [3]. If one
assumes that the exogenous inputs
Ii,
Ji
(i
=
1,2,
. .
.
,
n)
and
the connection weights
mij
(i,
j
=
1,2,
.
.
.
,
n)
are constants
while the neuronal output signal function
S
is a differentiable,
monotonic nondecreasing real valued function on
(-m,
m),
then it is possible to introduce an energy function (or Lyapunov
function)
E
such that
n
CT.
nn
n n
n
i=l
j=1
j=1
(2)
where
S’(z)
=
F.
It has been shown in
[9]
that
n
n
kl
j=1
One can show from (3) that as
t
i
m,
ii(t)
i
0,
yi(t)
+
0,
i
=
1,2,
. .
.
,
n
implying that the network (1) converges to an
equilibrium corresponding to the constant extemal inputs
Ii,
Ji
(i
=
1,2,
.
.
.
,
n).
The equilibria are sometimes called pattems
or memories associated with the extemal inputs
I
and
J.
Manuscript received
June
25, 1992; revised
June
15, 1993.
The
authors
are
with the School
of
Information Science
&
Technology,
IEEE
Log
Number
921 1596.
Flinders University, Adelaide,
SA
5001,
Australia.
It is possible to simplify bidirectional networks of the type
in
(1)
to a single system of a network of the type
for suitably defined nonlinear functions
fi,
i
=
1,2,
. .
.
,
n.
In fact, a referee suggested that we do such a simplification.
The authors have to retain the model
(1)
as it stands since
such a simplification will alter the bidirectional interplay of
the input-output nature of the two layers of the system and
will reduce the system to that of a single layer system. For
a detailed investigation of single layer systems we refer to a
recent article of Gopalsamy and He [7].
The purpose of this brief article is to investigate the exis-
tence and stability characteristics of the equilibria of networks
of the form
n
dUi(t)
-
dt
-
-ui(t)
+
j=1
UijS(AjWj(t
-
CrZj))
+
I;
i=
1,2,*..,n
(4)
in which
Xj
pj,
rij
,
aij
(2,
j
=
1,2,
.
.
.
,
n)
are nonnegative
constantsand&,
Ji,
ai;,
bij(i,j=
1,2,.‘.,n)
arerealnum-
bers; for convenience of exposition in the following we choose
the signal response function as follows
S(z)
=
tanh(x),
z
E
(-m,
00).
(5)
The time delays
T~~
and
~7%~
correspond to the finite speed of
the axonal transmission of signals; for example
T~,
corresponds
to the time lag from the time the 2-th neuron in the I-layer
emits a signal and the moment this signal becomes available
for the j-th neuron in the J-layer of (4) (see for instance
Domany
et
al.
[4]).
The constants
A,,
pJ
correspond to the
neuronal gains associated with the neuronal activations. We
refer to Babcock and Westervelt [l], Marcus and Westervelt
[12], [13] and Marcus
et
al.
[14] for linear analyses of single
layer networks with delays.
One of the problems in the analysis of the dynamics of the
delay differential system
(4)
is the existence
of
solutions of
(4).
The initial conditions associated with
(4)
are assumed to
be
of the form
u,(s)
=
h(s),
s
E
[-~*,0],
T*
=
maxlsz,23snTz,
u,(s)
=
&(s),
s
E
[-0*,0],
(T*
=
maxl<z,,<n
gZI
1
i
=
1,2,...,n
(6)
1045-9227/94$04.000 1994 IEEE
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL.
5,
NO.
6,
NOVEMBER
1994
999
in which
q5i,
$i
are continuous real valued functions defined
on their respective domains. One can use the method of steps
and continuation (see Elsgolt's and Norkin
[SI)
and show that
solutions of (4)-(6) exist for all
t
2
0.
The following result
provides sufficient conditions for the existence of equilibria
associated to each pair of inputs
I
and
J
in (4).
where
UljS(XjYj)
+
I1
cj=1
@jS(XjYj)
+
12
anjS(Xjyj)
+
In
bljqpjzj)
+
J1
Ej=1
bZjS(PjZj)
+
Jz
F~(x,y)
=
r='
...
1,
i) the connection weights
uij,
bij
(i,
j
=
1,2,.
. .
,
n)
are real
FZhY)
=
I
...
Theorem
1:
Assume the following:
constants;
ii) the exogenous inputs
Ii,
Ji
(i
=
1,2,.
..
,n)
are real
constants;
We note from (12) and (13) that if
(x,
y)
and
(X,
Y)
are any
two points of
R,
then
iii) the gain parameters
pj,
Xj
(j
=
1,2,
. . .
,
n)
and the time-
delays
rij
and
'~ij
(i,j
=
1,2,-..,n)
are
nonnegative
constants: llF(x9 Y)-F(X,Y)II
nn
=
I
[.ij{S(XjYj/j)
-
sj(xjY,)l
iv) there exists a number
c
E
(0,l) such that the neuronal
i=l
j=1
gains and the connection weights satisfy
+bij{S(Pjxj)
-
S(PjXj))l
Then corresponding to each exogenous input pair of vectors
I
=
(11,12,-.-,In) and
J
=
(J1,Jz,..-,Jn) there exists an
unique equilibrium
(U*,
'U*)
of
(4)
satisfying
n
n
i
=
1,2,.
.
.
,
n.
(8)
Uf
=
u;js(xj'U;)
+
Ii
j=1
W:
=
bijS(pju5)
+
Ji
j=1
Pro08
We have from (4) and
(5)
that an arbitrary solution
of (4)-(6) satisfies the following differential inequalities
i=1,2,...,n
(9)
-Ui(t)
-
a;
5
-wi(t)
-
pi
5
*
I
-ui(t)
+
a2
q
5
-wi(t)
+
pi
where
i
=
1,2,-..,n. (10)
Pi
=
lbijl
+
IJil
j=1
is invariant with respect to the delay differential equations
(4). Thus if the system (4) has an equilibrium, then such an
equilibrium is a fixed point of the mapping
F
:
R
+
RZn
defined by
nn
n
in deriving
(15)
and subsequent inequalities we have used the
facts that
8;
lies between
Xjyj
and
Xjq,
6'7
lies between
pjxj
and
pjXj
as well as
S'(8)
=
1
-
Sz(6)
5
1
for
8
E
(0,~).
The mapping
F
is continuous and
F(R)
C
R;
it follows from
(16) and c
<
1
that
F
is a contraction on
0.
By the well
known contraction mapping principle, we conclude that there
exists a unique point say
(U*,
'U*)
such that
(18)
F(u*,
w*)
=
(U*,
'U*)
and this completes the proof.
Thus there exists a unique pattem or memory (or equilib-
rium) associated with each set of the external inputs
I
and
J
when the connection weights
uij
and
bij
are fixed. In the next
section we derive conditions for the global asymptotic stability
of the unique equilibrium
(U*,
'U*)
of (4).
loo0
IEEE
TRANSACTIONS
ON
NEURAL
NETWORKS,
VOL.
5,
NO.
6,
NOVEMBER
1994
11.
RECALL
OF
ASSOCIATIVE MEMORY
The equilibrium or the pattem
(u*,w*)
of
(4)
associated
with a given
(I,
J)
is said
to
be globally asymptotically
stable independent of the delays if every solution of
(4)
corresponding to an arbitrarily given set of initial values
(6)
satisfy
solutions of
(21)
laijlXjlYj(t
-
UiJI
i=l
j=1
i=l
-
IYi(t)l
+
IbijIPMt
-
%)I
lim
U;@)
=U:
and lim
vi(t)
=
wz*
(i
=
1,2,.-.,n).
t-oo
t-co
(19)
Weremark that when
aij
#
bij,
Xj
#
1,
pj
#
1,
rij
0,
the method
of
Lyapunov functions exploited by Hopfield
[8],
Cohen and Grossberg
[3],
Kosko [9] is not applicable;
the existence of a Lyapunov function is dependent on the
symmetry of the synaptic connection weights in [8], [9] and
Theorem
2
Suppose the assumptions of Theorem 1 hold.
Then the equilibrium
(U*,
w*)
of
(4)
is globally asymptotically
stable.
0,
U
a..
-
b..
i
z3
-
z3
n the case of Kosko [9].
Proof:
Define the variables
xi
and
y;
by the following
j=1
n
One can derivative that the deviations
xi,
y;
are governed by
it is consequence of
(25)
that
It follows
from
(26)
and
(23)
that
V
is bounded on
(0,m);
the
boundedness of
V
on
(0,~)
implies that
xi,
yi
are bounded
(0,
CO)
and this implies that
xi,
y;
are uniformly continuous
on
(0,
CO).
We note from
(22)
that
ki,
$i
are bounded on
on
(0,~).
A
consequence of
(26)
is that
9
=
-yi(t)
+
E
b;jpjSi(BG(t))xj(t
-
~ij)
j=1
i
=
1,2,
".,n
(21)
in which
erj(t)
lies between
Xjvg
and
Xjvj(t-gij)
and
19:~(t)
lies between
pju;
and
pjuj(t
-
T~~)
for
j
=
1,2,.
. .
,
n.
We
note that
i=
1
S'(s)
=
1
-
S2(s)
5
1,
s
E
R
The uniform continuity
of
[Ixi(t)l
+
Iyi(t)l]
on
(0,~)
together with
(27)
implies by Barbalat's Lemma (see Barbalat
and hence we have from
(21)
that
n
[2]
or Gopalsamy
[6])
that
n
n
[Ixi(t)l+
~yi(t)ll
-+
o
as
t
+
00.
(28)
i=l
Thus it follows
xi(t)
--+
O,yi(t)
-+
0
as
t
-+
CO,
i
=
1,2,
. .
.
,
n
and hence
(22)
9
5
-xi(t)
+
E
lUijIXjlYj(t
-
%j)(
3=1
5
-yi(t)
+
Ibij(pjlzj(t
-
~j)l
j=1
i
=
1,2,...,n.
Consider a Lyapunov functional
V(t)
=
V(z,
y)(t)
defined by
L
111.
DISTRIBUTED DELAYS
The use of constant fixed delays in models of delayed
feedback provides of a good approximation in simple circuits
consisting
of
a small number of cells. Neural networks usually
have a spatial extent due to the presence of a multitude of
(23)
I
+
2
IbijlPj
J'
IZj(S)I
ds
*
3=l
t-rq
Calculating the upper right derivative
D+V
of
V
along the
IEEE
TRANSACTIONS
ON
NEURAL
NETWORKS,
VOL.
5,
NO.
6,
NOVEMBER
1994
1001
parallel pathways with a variety of axon sizes and lengths.
Thus there will be a distribution of conduction velocities along
these pathways and a distribution of propagation delays. In
these circumstances the signal propagation is not instantaneous
and cannot be modeled with discrete delays and a more
appropriate way is to incorporate continuously distributed
delays. The extent to which the values of the state variable in
the past affect its present dynamics is determined by a delay
kemel; the case of constant discrete delays corresponds to a
choice of the delay kemel to
be
a Dirac delta function. Kernels
of the form
p+l
m!
K(t)
=
-tme--at
,
t
2
0,
a,m
E
[O,m)
have been found mathematically tractable in many models
of dynamic systems in mathematical ecology especially in
population dynamics.
We shall now consider a class of bidirectional associa-
tive memory networks with continuously distributed delays
described by
dt
7=1
03
I
in which the extemal inputs
li,
Ji
and the connection weights
aij
,
bij
are constants; the neuronal gains
Xj
and
pj
are positive
constants; the delay kemels
IC::),
k$)
are nonnegative valued
continuous functions defined on
[0,
m)
satisfying
The initial values associated with (29) are of the foq
Ui(S)
=
di(S),
Yi(S)
=
$i(S),
s
E
(-,OI,
i=l,2,...,n (31)
where
di,
$;
are assumed
to
be bounded continuous functions
defined on
(-w,O].
For an extensive discussion of the stability
and asymptotic behavior of integro-differential equations such
as (29) and their applications, we refer to the recent monograph
by Gopalsamy [6]. For applications of integro-differential
equations with continuously distributed delays such as those
in (29), we refer to Tank and Hopfield [15].
One can see that the equilibrium
(U*,
w*)
defined by (15) is
again an equilibrium of (29) due to (30).
Theorem3:
Suppose the hypotheses of Theorem 1 hold:
suppose further
(30)
holds. Then the equilibrium
(U*,
U*)
of
(29) is globally asymptotically stable in the sense that all
solutions of (29)-(31) satisfy
Proof:
Details of proof are similar to those of Theorem 2
and hence we shall to brief. As before we define
xi,
yi
by (22)
and derive from (29)-(30) that
zi
and
yi
satisfy the system
which as before leads to
i
=
1,2,...,n. (34)
We consider a Lyapunov type functional
V(t)
=
V(x,y)(t)
defined by
i=l
(35)
By our assumption on the initial values on
(-m,O]
and the
hypotheses on the delay kemels, one can verify that
V
is
defined on
(0,m)
and
V
is bounded on
(0,
m).
Also the upper
right derivative
D+V
of
V
along the solutions of (33) can be
calculated
so
that
n
D+V(t)
I
-(I
-
c)
[IZi(t)l
+
IYZ(t)ll.
(36)
i=l
The remaining details of proof are identical to these of
Theorem 2 and hence are omitted. The proof is complete.
IV.
REMARKS
Recall
of
memories is one of the processes by which the
brain retums in some sense from a current state to another state
in which it has been before. In neural network models, memory
corresponds to a temporally stationary or nonstationary equi-
librium and recall is modeled by the convergence of neuronal
activations in the neuronal activation space to the equilibrium.
The trigger provided for the system to recall the memory may
come from outside as extemal inputs. Thus the patterns or
equilibria associated with extemal inputs are recalled by the
convergence of system dynamics; global asymptotic stability
of an equilibrium means that the recall is “perfect” in the
sense no hints or guesses are needed as in the case of local
stability analyses; that is when the extemal inputs are provided
to the system, irrespective
of
the initial values, the system
converges to the equilibrium associated with the inputs. Recall
1002
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL.
5,
NO.
6,
NOVEMBER
1994
with the help of hints and guesses correspond to local stability
of
equilibria; since the initial values have to
be
in a suitable
neighborhood of the corresponding equilibrium.
It is known (see
[12])
that time delays in response or
transmission can induce sustained oscillations and “chaos”. We
have discussed the stability characteristics
of
an already trained
bidirectional not necessarily symmetric associative network
with transmission delays and obtained sufficient conditions for
[5]
L.
E. El’sgolt’s and
S.
B. Norkin,
Introduction to The Theory and
Application of Differential Equations with Deviating Arguments.
New
York
Academic Press, 1973.
[6]
K.
Gopalsamy,
Stabilify and Oscillations in Delay DifSerential Equa-
tions of
Population Dynamics.
Dordrecht,
The
Netherlands: Kluwer
Academic, 1992.
171
K.
Gopalsamy and
X.
Z.
He, “Stability in asymmetric Hopfield nets
with transmission delays,”
Physica D
(to appear).
tional properties like those of two state neurons,”
P
roc. Nut. Acad. Sci.
181
J.
Hopfield, “Neurons with graded response have collective computa-
the absence of delay induced persistent oscillations or chaos.
USA,
vol.
81,
pp. 3088-3092, 1984.
In
a
forthcoming
article
we
discuss
the
dynamics
of
associative
recall in a bidirectional adaptive network with delays in the
learning dynamics.
[9] B. Kosko, “Bidirectional associative memories,”
IEEE. Trans. Syst. Man
and Cybernetics,
vol. SMC-18, pp. 49-60, 1988.
vol. 26 pp. 49474960, 1987.
vol.
1,
pp. 44-57, 1990.
1101
-
,
“Adaptive bidirectional associative memo,.ies,”
~~~1.
Oprjcs.,
[
111
-,
“Unsupervised learning in noise,”
IEEE.
Trans. Neural Net.,
1121 C. M. Marcus and
R.
M. Westervelt, “Dvnamics of analoe. neural
REFERENCES
[l] K.
L.
Babcock and R. M. Westervelt, “Dynamics of simple electronic
neural networks,”
Physica,,
vol. 28D, pp. 305-316, 1987.
[2]
I.
Barbalat, “Systemes d’equations differentielle d’oscillations nonlin-
eaires,”
Rev. Roumaine Math. Pures Appl.,
vol. 4, pp. 267-270, 1959.
[3] M. A. Cohen and
S.
Grossberg, “Absolute stability and global pattern
formation and parallel memory storage by competitive neural networks,”
IEEE. Trans. Syst. Man. and Cybernetics,
vol. SMC-13
,
pp. 815-821,
1983.
[4] E. Domany,
J.
L.
van Hemmen, and K. Schulten, ed.,
Models
of
Neural
Networks.
Berlin, Germany: Springer-Verlag, 1991.
~~
networks with time delay,” in
Advances in Neural Information Proc.,
San Menlo, CA, 1989, pp. 568-576.
1131
-,
“Stability of analog neural networks with time delay,”
Phys.
Rev. A,
vol. 39, pp. 347-359, 1989.
1141
C.
M. Marcus,
F.
R. Waugh, and
R.
M. Westervelt, “Nonlinear dy-
namics and stability of analog neural networks,”
Physica,
vol. 51D, pp.
234-247, 1991.
[
151 D. W. Tank and
J. J.
Hopfield, “Neural computation by concentrating
information in time,”
Proc. Nut. Acad.
Sci.
USA,
vol. 84, pp. 1896-1991,
1987.
I.