Showing papers on "Random neural network published in 1998"

Fine structure of neural spiking and synchronization in the presence of conduction delays (hippocampusyg-rhythmsydoublets)

Abstract: Hippocampal networks of excitatory and in- hibitory neurons that produce g-frequency rhythms display behavior in which the inhibitory cells produce spike doublets when there is strong stimulation at separated sites. It has been suggested that the doublets play a key role in the ability to synchronize over a distance. Here we analyze the mechanisms by which timing in the spike doublet can affect the synchro- nization process. The analysis describes two independent effects: one comes from the timing of excitation from sepa- rated local circuits to an inhibitory cell, and the other comes from the timing of inhibition from separated local circuits to an excitatory cell. We show that a network with both of these effects has different synchronization properties than a net- work with either excitatory or inhibitory type of coupling alone, and we give a rationale for the shorter space scales associated with inhibitory interactions. When neurons communicate over some distance, there are conduction delays between the firing of the presynaptic neuron and the receipt of the signal at the postsynaptic cell. It is also known that cells can synchronize over distances of at least several millimeters, over which conduction delays can be significant. This raises the question of how cells can synchro- nize in spite of the delays. Traub et al. (1) and Whittington et al. (2) suggested that the fine structure of the spiking of some of the cells may play a part in the synchronization process for the g frequency rhythm, found in hippocampal and neocortical systems during states of sensory stimulation. (For references, see ref. 2.) More specifically, for some models of cortical structure, they noted that the ability to synchronize in the presence of delays is correlated with the appearance of spike doublets in the inhibitory cells. The doublets appear in slice preparations when there is strong stimulation at separated sites (1, 2). In this paper, we analyze a mechanism for such synchronization, using a simplified version of equations of Traub and colleagues. The timing of spikes within a doublet is shown to encode information about phases of local circuits in a previous cycle; the model shows how the circuit can use this information in an automatic way to bring nonsynchronous local circuits closer to synchrony. There are two independent effects in the model. The first is the response of the inhibitory (I) cells to excitation from more than one local circuit. The I-cells may produce more than one spike, whose relative timing depends on strength of excitation and recovery properties of the cell after the firing of a first spike; the latter can include effects of after- hyperpolarization or self-inhibition in a local circuit. The second effect is the response of the excitatory (E) cells to the multiple inhibitory spikes they receive from within their local circuit or other circuits. The maximal inhibition received by an E-cell can depend on the times and sizes of the inhibitory postsynaptic potentials it receives, and this affects the time until the E-cell can spike again. We show that each of the two effects is enough to allow synchronization. Together, they give the network synchronization properties that are not intuitively clear from the properties of either alone. Previous papers have analyzed mechanisms for synchroni- zation depending on interactions among I-cells (3-6) or E-cells (5-10). In this paper, the interactions between the local circuits include E3 I and I3 E. We omit the E3 E connections, which are sparse in the CA1 region of the hippocampus (11), and consider only those I3 I connections that are sufficiently local to be considered part of a local circuit. By considering networks with a subset of these connections, we shed light on the role of each of them in the synchronization process. In particular, we show that the different kinds of coupling work together to provide synchrony over a larger range of delays than either could do alone, and that the interaction provides a significant increase in the speed of synchronization. The I3 E coupling also helps provide robustness to disruption from larger excitatory conductances, but it reduces robustness to heterogeneity. The two effects together give a rationale for the shorter space scales of the inhibitory interactions. (See Discus- sion.) Our analysis considers a pair of local circuits, each having one E-cell and one I-cell; each cell represents populations of neurons. We reduce the biophysical equations for the network to a map that takes the interspike interval of the two excitatory cells to a new interspike interval after one cycle. The map does not depend on the details of the biophysical equations. (In the motivating equations in the Appendix, each cell has basic Hodgkin-Huxley-like spiking currents.) The map is derived from two subsidiary maps that encode the times that an inhibitory cell or an excitatory cell fires after receiving inputs at two different times as a function of the time difference between the inputs. From these maps, we are able to read off information about how different kinds of coupling affect stability of the synchronized state, the period of the synchro- nized solution, the rate of synchronization, and the response of the network to heterogeneity of the cells. The importance of multiple spikes in the synchronization process distinguishes the mechanisms of this paper from other mechanisms of synchronization that deal with the envelope of bursting activity (3, 9) or single pulses (5-8). Indeed, the significance of the timing of individual spikes provides a new aspect of ''temporal coding''; the spikes encode information about the synchronization process, rather than information directly related to sensory inputs.

On the optimal number of hidden nodes in a neural network

Abstract: In this study we show, empirically, that the best performance of a neural network occurs when the number of hidden nodes is equal to log(T), where T is the number of training samples. This value represents the optimal performance of the neural network as well as the optimal associated computational cost. We also show that the measure of entropy in the hidden layer not only gives a good foresight to the performance of the neural network, but can be used as a criteria to optimize the neural network as well. This can be achieved by minimizing the network entropy (i.e. maximizing the entropy in the hidden layer) as a means of modifying the weights of the neural network.

Image and video compression

Abstract: The authors discuss the underlying principles of image and video compression. The network model they use for image compression is the random neural network (RNN). This pulsed network model provides a somewhat more accurate representation of what occurs in "real" neurons. Signals in the form of pulse trains travel between neurons. These pulses can be either excitatory (we call these "positive" pulses), or they can be inhibitory or "negative". Just like many naturally occurring neural nets, these pulses all have the same magnitude which is normalized as 1. A neuron in the RNN emits pulses at an instantaneous rate proportional to its degree of excitation and to its rate of firing. Besides being more accurate, the RNN is also useful because an algorithm, which allows for the training of a fully recurrent RNN, has been designed. This means it is possible to find good weights between neurons even if every neuron has a connection to every other neuron. This full recurrence is not easily allowed in standard back propagation networks.

Resolution of pattern recognition problems using a hybrid Genetic/Random Neural Network learning algorithm

Abstract: Gelenbe has proposed a neural network, called a Random Neural Network, which calculates the probability of activation of the neurons in the network. In this paper, we propose to solve the patterns recognition problem using a hybrid Genetic/Random Neural Network learning algorithm. The hybrid algorithm trains the Random Neural Network by integrating a genetic algorithm with the gradient descent rule-based learning algorithm of the Random Neural Network. This hybrid learning algorithm optimises the Random Neural Network on the basis of its topology and its weights distribution. We apply the hybrid Genetic/Random Neural Network learning algorithm to two pattern recognition problems. The first one recognises or categorises alphabetic characters, and the second recognises geometric figures. We show that this model can efficiently work as associative memory. We can recognise pattern arbitrary images with this algorithm, but the processing time increases rapidly.

A dynamic neural network with temporal coding and functional connectivity

Abstract: A neural network model capable of altering its pattern classifying properties by program input is proposed. Here the “program input” is another source of input besides the pattern input. Unlike most neural network models, this model runs as a deterministic point process of spikes in continuous time; connections among neurons have finite delays, which are set randomly according to a normal distribution. Furthermore, this model utilizes functional connectivity which is dynamic connectivity among neurons peculiar to temporal-coding neural networks with short neuronal decay time constants. Computer simulation of the proposed network has been performed, and the results are considered in light of experimental results shown recently for correlated firings of neurons.

Probability density methods for smooth function approximation and learning in populations of tuned spiking neurons

Abstract: This article proposes a new method for interpreting computations performed by populations of spiking neurons. Neural firing is modeled as a rate-modulated random process for which the behavior of a neuron in response to external input can be completely described by its tuning function. I show that under certain conditions, cells with any desired tuning functions can be approximated using only spike coincidence detectors and linear operations on the spike output of existing cells. I show examples of adaptive algorithms based on only spike data that cause the underlying cell-tuning curves to converge according to standard supervised and unsupervised learning algorithms. Unsupervised learning based on principal components analysis leads to independent cell spike trains. These results suggest a duality relationship between the random discrete behavior of spiking cells and the deterministic smooth behavior of their tuning functions. Classical neural network approximation methods and learning algorithms based on continuous variables can thus be implemented within networks of spiking neurons without the need to make numerical estimates of the intermediate cell firing rates.

Distributed, Event Driven Simulation of Spiking Neural Networks.

Abstract: We present the architecture of a simulator that is able to simulate large networks of spiking neurons using a distributed event driven simulation. Contrary to a time driven simulation, which is usually used to simulate spiking neural networks, our simulation needs less computational resources because of the low average activity of typical networks. The simulator is divided into a set of communicating sub-simulators running concurrently on several workstations, where each sub-simulator handles a part of the network. The paper addresses the problems of synchronisation between the sub-simulators and how information contained in the network topology and properties of neuron models are used to solve them. Preliminary results are presented for two simple model networks illustrating the speed up gained by a distribution of the simulation.

The coding of information by spiking neurons : an analytical study

Abstract: We analyse analytically the coding of information by a spiking neuron. The emphasis is on the question of how many spikes are necessary for the reliable discrimination of two different input signals. The discrimination ability is measured by the second-order Renyi mutual information between the random variable describing the name of the signal and a sequence of n output spikes. Analysing this measure as a function of n, we study the coding strategy of a single spiking neuron, with the following main results. A small number of output spikes is required for efficient discrimination of input signals, i.e. for encoding them, if the separation is easy; a large number of output spikes is required in the difficult case of separation of very similar input signals. Three different versions of the spike response model of a single neuron are studied. The approach presented can be regarded as a non-parametric version of the reconstruction method of Bialek.

Random neural network recognition of shaped objects in strong clutter

Abstract: Detecting objects in images containing strong clutter is an important issue in a variety of applications such as medical imaging and automatic target recognition. Artificial neural networks are used as non-parametric pattern recognizers to cope with different problems due to their inherent ability to learn from training data. In this paper we propose a neural approach based on the Random Neural Network model (Gelenbe 1989, 1990, 1991, 1993), to detect shaped targets with the help of multiple neural networks whose outputs are combined for making decisions.

Totally model-free reinforcement learning by actor-critic Elman networks in non-Markovian domains

Abstract: We describe how an actor-critic reinforcement learning agent in a non-Markovian domain finds an optimal sequence of actions in a totally model-free fashion; that is, the agent neither learns transitional probabilities and associated rewards, nor by how much the state space should be augmented so that the Markov property holds. In particular, we employ an Elman-type recurrent neural network to solve non-Markovian problems since an Elman-type network is able to implicitly and automatically render the process Markovian. A standard "actor-critic" neural network model has two separate components: the action (actor) network and the value (critic) network. In animal brains, however, those two presumably may not be distinct, but rather somehow entwined. We thus construct one Elman network with two output nodes: actor node and critic node, and a portion of the shared hidden layer is fed back as the context layer, which functions as a history memory to produce sensitivity to non-Markovian dependencies. The agent explores small-scale three and four-stage triangular path-networks to learn an optimal sequence of actions that maximizes total value (or reward) associated with its transition from vertex to vertex. The posed problem has deterministic transition and reward associated with each allowable action (although either could be stochastic) and is rendered non-Markovian by the reward being dependent on an earlier transition. Due to the nature of neural model-free learning, the agent needs many iterations to find the optimal actions even in small-scale path problems.

Method and apparatus for generating a model of an industrial production

Abstract: A process model of an industrial process or system is generated. The model correlates a first number M of process parameters forming input values with a second number L of quality characteristics forming output values, which are processed to form feedback control signals for the process or system. A third number N of training data sets of the industrial process are first gathered and processed during a learning phase of the model with the help of a central processing unit, whereby a preliminary approximately model is used including a neural network with local approximation characteristics. The neural network is connected in parallel with a linear network. Both networks are connected to the same inputs. The neural network initially has a number N of neural cells corresponding to the number of training data sets. A weighted linear combination of the M process parameters is performed. The linear network and the neural network are connected with their outputs through weighting circuits to a common summing point. A stepwise regression is performed to reduce the number of neural cells from N to K and of linear paths from M to M-R. Closed loop feedback signals control the industrial process.

The spiked random neural network: nonlinearity, learning and approximation

Abstract: We summarize the theoretical foundations of the random neural network model (RNN) and of its learning algorithm, and present a relevant bibliography of its theory and applications. Many applications have resulted from this model, including its use in still image and video compression which has achieved compression ratios of up to 500:1 for moving gray-scale images, with 30db PSNR quality levels. Another application of the RNN is to image segmentation; the recurrent feature of the network has been used to extract precise morphometric information from magnetic resonance imaging (MRI) scans of the human brain. The RNN has also been successfully applied to optimization and image texture analysis and reconstruction.

The Evolution of a Feedforward Neural Network trained under Backpropagation

Abstract: This paper presents a theoretical and empirical analysis of the evolution of a feedforward neural network (FFNN) trained using backpropagation (BP). The results of two sets of experiments axe presented which illustrate the nature of BP’s search through weight space as the network learns to classify the training data. The search is shown to be driven by the initial values of the weights in the output layer of neurons.

A locally inhibited lateral neural network: on the fundamental features of a network consisting of neurons with a restricted range of interaction.

Abstract: A network model that consists of neurons with a restricted range of interaction is presented. The neurons are connected mutually by inhibition weights. The inhibition of the whole network can be controlled by the range of interaction of a neuron. By this local inhibition mechanism, the present network can produce patterns with a small activity from input patterns with various large activities. Moreover, it is shown in simulation that the network has attractors for input patterns. The appearance of attractors is caused by the local interaction of neurons. Thus, we expect that the network not only works as a kind of filter, but also as a memory device for storing the produced patterns. In the present paper, the fundamental features and behavior of the network are studied by using a simple network structure and a simple rule of interaction of neurons. In particular, the relation between the interaction range of a neuron and the activity of input-output patterns is shown in simulation. Furthermore, the limit of the transformation and the size of basin are studied numerically.

Multipoint routing and traffic management in high-speed networks

Abstract: High speed networking has brought with it a number of challenges due mainly to the need for supporting applications having diverse quality of service (QoS) requirements. QoS is typically characterized by parameters such as loss and end-to-end latency. The asynchronous transfer mode (ATM) technology, designed to be scalable in speed and geographical distance, has the ability to seamlessly integrate data from various applications for transport over a single network. As such, it is widely considered to be the transport technology of choice for emerging and future high speed networks such as the broadband integrated services digital network (B-ISDN). In this work, we are concerned with the key areas of multiparty communications and traffic control mechanisms for high speed networks with an emphasis on ATM. Multicast communication involves the transport of data to multiple receivers. In order to enable multicast data transfer in an ATM network, a tree must be constructed which spans the source and all the destinations. For the purpose of routing, the network is usually modeled as a weighted, undirected graph. The edge weights represent the cost to be optimized while constructing the tree. The problem is to find a minimum Steiner tree for the graph given a set of destinations. This dissertation reviews available heuristics for solving this problem which run in polynomial time. We then use the random neural network (RNN) to improve on the solutions delivered by these heuristics. Next, we consider a different flavor of the problem for delay sensitive applications. Here, the goal is to minimize tree cost while ensuring a bounded end-to-end delay between the source and each of the destinations and requires the construction of a constrained minimum Steiner tree. We review existing heuristics for this problem, and then develop a new one. For both of the above problems, exhaustive simulation shows that the new heuristics are able to find trees that are significantly closer to optimal than those found by the existing ones in many instances. Our next concern is a method to provide efficient multicast support for large multicast connections in an ATM network. In order to support multicast, ATM switches at the branch-point of a multicast tree must be capable of replicating cells from an input port and routing them to multiple output ports. We develop a queueing model to study this behavior in a single node. Finally, we explore issues concerning traffic management in ATM networks. We review existing work on scheduling disciplines ranging from simple priority schemes to weighted fair queueing. We then focus on a dynamic priority queueing method for multiple classes of traffic in an ATM network. This scheduling discipline has a very desirable property of providing minimum bandwidth guarantees for each class of traffic. We use an approximation to perform a simple queueing analysis for this system. We find that the approximation yields very accurate results for a variety of traffic conditions and operating parameters. (Abstract shortened by UMI.)