Stochastic automata operating in an unknown random environment have been proposed earlier as models of learning. These automata update their action probabilities in accordance with the inputs received from the environment and can improve their own performance during operation. In this context they are referred to as learning automata. A survey of the available results in the area of learning automata has been attempted in this paper. Attention has been focused on the norms of behavior of learning automata, issues in the design of updating schemes, convergence of the action probabilities, and interaction of several automata. Utilization of learning automata in parameter optimization and hypothesis testing is discussed, and potential areas of application are suggested.

/pdf/learning-automata-a-survey-48tr2mi6qw.pdf

Learning Automata - A Survey

A probabilistic or stochastic automaton (pa) is a device with a finite number of internal states that scans input words over a finite alphabet and responds by successively changing its state in a probabilistic way.

Probabilistic automata

This paper solves the classical two-armed-bandit problem under the finite-memory constraint described below. Given are probability densities p_0 and p_1 , and two experiments A and B . It is not known which density is associated with which experiment. Thus the experimental outcome Y of experiment A is as likely to be distributed according to p_0 as it is to be distributed according to p_1 . It is desired to sequentially choose an experiment to be performed on the basis of past observations according to the algorithm T_n = f(T_{n-1}, e_n, Y_n), e_n = e(T_{n-1}) , where T_n \in \{1, 2, \cdots, m\} is the state of memory at time n, e_n \in \{A, B\} is the choice of experiment, and Y_n , is the random variable observation. The goal is to maximize the asymptotic proportion r of uses of the experiment associated with density p_0 . Let l(y) = p_0 (y) / p_1 (y) , and let \bar{l} and \bar{\bar{l}} denote the almost everywhere greatest lower bound and least upper bound on l(y) . Let 1 = \max {\bar{\bar{l}}, 1/\bar{l}} . Then the optimal value of r , over all m -state algorithms (f, e) , will be shown to be l^{m-1} / (l^{m-1} + 1) . An e -optimal family of m -state algorithms will be demonstrated. In general, optimal algorithms do not exist, and e -optimal algorithms require artificial randomization.

/pdf/the-two-armed-bandit-problem-with-time-invariant-finite-5794qo3bi3.pdf

The two-armed-bandit problem with time-invariant finite memory

A class of nonstationary environments with unknown but periodically changing probabilistic characteristics is considered. It is proposed to optimize the performance in an environment from this class by using a two-level system of variable-structure stochastic automata. The first level estimates the unknown period while the second level operates suitably in the environment for one cycle assuming that this estimate is the true period of the environment. The average output of the environment in this cycle is used as the input to the first level to determine the next estimate of the period. The optimal performance of this two-level system of automata in periodic random environments is demonstrated through computer simulations.

A two-level system of stochastic automata for periodic random environments

We propose a new nonstationary random environment R(C1(t,?),...,Cr(t,?)), where t represents time and ? ? ?, ? being the supporting set of a probability measure space (?,B,?). Moreover, the learning performance of the Lr-1 scheme under R(C1(t,?),..., Cr(t,?)) is discussed.

On the Learning Behavior of Stochastic Automata Under a Nonstationary Random Environment

This paper considers the effect of environmental nonstationarities on the performance of a stochastic automaton when it is used in the synthesis of an adaptive controller. The stochastic automaton considered has a variable structure in that its state probabilities are continuously altered according to a reinforcement scheme in response to penalties received from the environment. The automaton adapts by reducing the average penalty. Periodic perturbations of penalty strengths are used as "test signals" to derive analytic expressions describing the "tracking" behavior of the automaton operating under a linear reinforcement scheme. The pertinent parameters governing the adaptive behavior are discussed in detail. Digital simulation studies are presented for 2-state and 10-state cases. The results agree with the theoretical analysis.

D. C. Shen

Papers

Adaptation of stochastic automata in nonstationary environments