Showing papers in "Information & Computation in 1963"

On context-free languages and push-down automata

Abstract: This note describes a special type of one-way, one-tape automata in the sense of Rabin and Scott that idealizes some of the elementary formal features used in the so-called “push-down store” programming techniques. It is verified that the sets of words accepted by these automata form a proper subset of the family of the unambiguous context-free languages of Chomsky's and that this property admits a weak converse.

Power moment identities on weight distributions in error correcting codes

Abstract: A series of identities relating the weight distribution in any code space to the weight distribution in the orthogonal code space has been given by Mrs. J. MacWilliams 1962. Here we derive a series of power moment identities from the MacWilliams identities. An earlier result of Assmus and Mattson is shown to be equivalent to the third power moment identity. A unique solution to the power moment identities is given under certain conditions. Applications are given.

Computation with finite fields

Abstract: A technique for systematically generating representations of finite fields is presented. Relations which must be physically realized in order to implement a parallel arithmetic unit to add, multiply, and divide elements of finite fields of 2n elements are obtained. Finally, techniques for using a maximal length linear recurring sequence to modulate a radar transmitter and the means of extracting range information from the returned sequence are derived.

Three theorems on phrase structure grammars of type 1

Abstract: It is shown that the class of languages generated by type 1 phrase structure grammars is not enlarged by allowing end markers, that this class is closed under the operation of intersection, and that those languages representable by linear bounded automata belong to this class.

Regularity preserving modifications of regular expressions

Abstract: This paper is concerned with the problem of determining whether a set of sequences R \t', obtained by some given rule from a regular set of sequences R , is again a regular set. A number of such problems are solved in this paper and a basic technique is used which is easy to apply to problems of this type. This technique yields an explicit construction of the sequential machine which recognizes the sequences in R \t', if R \t' is a regular set. Among other things it is shown that: (i) the derivative of a regular set R with respect to any set of sequences W is a regular set, although a regular expression designating this set cannot in general be computed; (ii) the set of sequences obtained from a regular set R by removing “arbitrary halves” of the sequences in R and the set of these removed “halves” are both regular sets; (iii) the set of sequences, obtained from a regular set R by making no more than k changes in any m consecutive digits in sequences from R , is regular; (iv) the set of sequences which can be concatenated in one and only one way from sequences in R is regular.

Phrase structure languages, finite machines, and channel capacity

Abstract: The concept of channel capacity is applicable and useful in the study of languages with structures more general than those generated by the finite state channels discussed in Information Theory. A technique has been developed in this paper whereby the channel capacity of a class of phrase structure languages can be calculated. It has been shown that certain self-embedding phrase structure languages can be described by means of a finite machine whose symbols are the transformations of the phrase structure grammar.

Affine m-ary gray codes

Abstract: This paper presents a nontable-lookup technique for generating a class of n-dimensional m-ary Gray codes for every pair (n, m) of integers. It does not seek to exhaust all such codes, but restricts itself to a class characterized by simple encoding and decoding schemes. The simplicity shows up not only in the mathematical description of these codes as affine transforms of the sequence of m-ary numbers, but also in the circuitry required for realizations. For instance, a serial decoder can be built of a single unit delay and a single modular adder.

A geometric test-synthesis procedure for a threshold device

Abstract: A Boolean function may be defined as a mapping from the vertices of an n -dimensional hypercube whose vertices are n -tuples of “1” and “−1” to (0, 1). A threshold function is then a Boolean functin whose “1” or “TRUE” vertices are separable from the “0” or “FALSE” vertices by a hyperplane. Using this geometric representation, one may show that a threshold device realization may be approximated as follows: Form the vector sum of the TRUE vertices. The components of the resultant vector can then serve as the weights. The threshold is approximated by 2 n −1 minus the number of TRUE vertices. If the function under consideration is a threshold function, the vector formed from the first approximation and the threshold may be repeatedly rotated until it converges on a position such that its components form an exact threshold device realization. A specific procedure for performing these rotations is given. If the Boolean function is not a threshold function, the procedure must oscillate. The procedure can thus be used both as a test and a synthesis procedure. Further, whether or not the function is a threshold function, the procedure can be stopped at any point to yield an approximation to a threshold device realization. The desirable features of this procedure are: (1), it is simpler than the usual procedure of solving a set of inequalities; and (2), it will yield approximate realizations.

Error-correcting codes: An axiomatic approach

Abstract: This paper emphasizes that coordinates of an error-correcting code are functions. We have incorporated certain ideas already present in the literature into an axiomatic treatment of error-correcting codes. The usual “coordinates” are functions defined on a set of messages, with values in a given set K (classically, K = 0, 1). These functions are required to satisfy a certain “Coding Axiom”. In Section II we treat linear codes, K now being a finite ring; and we also give a treatment of cyclic codes, where we prove that the minimum distance is at least n/k , where n is the block length and k the dimension. In Section III we prove our main result, the Mapping Theorem, which relates the weights in a code A to those in an image-code B ; image-codes are defined simply in our terms. A number of corollaries are noted, including a formula for the sum of the squares of the weights of all code elements.

Reduction of feedback loops in sequential circuits and carry leads in iterative networks

Abstract: Techniques are presented for making use of “previous≓ inputs and outputs in designing sequential circuits and iterative networks. Theorems are proved regarding the maximum reduction in feedback loops or carry leads which can be obtained by the use of such techniques. An algorithm for testing whether a given state table corresponds to a definite event results as a by-product of the more general techniques presented here.

A class of convolution codes

Abstract: This paper describes a class of infinite convolution codes which are designed to minimize the time required to recover from an erasure burst on the binary erasure channel. It is shown that for any given rate, there exists a unique optimum erasure burst correcting code. For rates of the restricted form R = n /( n + 1), an algorithm is given by which the code of this rate may be written down by inspection. For other rates, the codes can be determined by a more complicated procedure based on evaluating large binary determinants.

The undecidability of the ambiguity problem for minimal linear grammars

Abstract: The ambiguity problem for general context-free phrase structure grammars has been shown undecidable. This paper proves the undecidability of a new form of Post's correspondence problem. Using this result, the undecidability of the ambiguity problem for minimal linear grammars is obtained.

A note on “very noisy≓ channels

Abstract: A “very noisy≓ channel is defined. This definition corresponds to many physical channels operating at low signal-to-noise ratio. For “very noisy≓ discrete input memoryless channels, the computation cutoff rate for sequential decoding, Rcomp, is shown to be one-half the capacity, C. Furthermore, that choice of input probabilities which achieves C also maximizes Rcomp, and vice versa.

Capacity and error bounds for a time-continuous Gaussian channel

Abstract: A model proposed by Fortet for a time-continuous Gaussian channel is analyzed. The model differs from that of Shannon in that a different constraint is imposed on the allowable input signals, and in addition the transmission of a code word in a given time interval is not assumed to interfere with the transmission of a word in any other interval. The capacity of the Fortet channel is established by proving a coding theorem and its weak converse. It is shown that the probability of error of an optimal code approaches zero exponentially with the time duration of the code words, provided the transmission rate is below channel capacity.

Entropy of Tamil prose

Abstract: The proportions of the different letters of the alphabet in Tamil prose are estimated from a large sample and an optimum code is constructed. The prose is compared with the Tamil poetry of different periods and the one-gram entropy of prose is significantly different from the entropies of the poetical works considered. An estimate is made, experimentally, of the entropy of Tamil prose.

On channels without a capacity

Abstract: which therefore always exists when C(k) exists. Many writers on information theory call by implication C(0-k) the capacity of the channel. They do this by proving a coding theorem and a weak converse (Wolfowitz, 1961, Section 7.6), and then calling the constant involved the capacity. In Wolfowitz (1961), where C is called the capacity, I pointed out that, to prove that C is the capacity, one has to prove a coding theorem and strong converse (Wolfowitz, 1961, Section 5.6, esp. p. 59).

A simple proof of the decipherability criterion of Sardinas and Patterson

Abstract: A simple alternative proof is given for a necessary and sufficient condition for the decodability of a sequence of codes. The proof involves no application of linear syntactical bases or other sophisticated algebraic criteria.

Redundancy and complexity of logical elements

Abstract: This paper deals with the problem of increasing the reliability of gate-type logical circuits through the use of redundancy. We will derive a lower bound on the amount of redundancy necessary to achieve a certain error correcting ability and show how this bound varies with the complexity of the elements used in the design of the redundant circuit, measured by the number of inputs. The complexity of encoders of block codes for transmission of information is defined. A bound similar to the one mentioned above on the error correcting ability of codes is derived which depends on the codes' rate of transmission and on the complexity of their encoders. Finally, we establish a connection between the bound on the error correcting ability of a redundant circuit and the bound on the error correcting ability of a block code.

On dynamic switching in one-dimensional iterative logic networks*

Abstract: A sequential iterative network (SITN) is a cascade of identical finite automata such that the i th automaton receives an x i input from the outside world and a y i input from its left neighbor, and produces a z i output to the outside world and a y i +1 output to its right neighbor. We prove three main theorems: ( 1 ) For every integer k there is a cell definition such that a corresponding SITN either can or cannot switch from equilibrium to a cycling condition (i.e., oscillation) following a single x i change according as n ≦ k or n > k , respectively; ( 2 ) there do not exist algorithms to tell whether a given cell definition admits of a SITN that can start from equilibrium and following a single x i change either (a) switch into a cycling condition, or (b) put out a z i = 1 during a switching transient; and ( 3 ) there do not exist algorithms to tell whether a given SITN cell definition must have every switching transient following a single x i change from equilibrium either (a) die out a bounded number of cells to the right of the change, or (b) extend all the way to the SITN boundary. All theorems are proved constructively on finite-state diagrams, and ( 2 ) and ( 3 ) hinge on an embedding of Minsky's Post Tag system results into such diagrams. We conclude with several iterative network equivalence demonstrations.

On the theory of linear dynamic system with periodic parameters

Abstract: This paper presents an extension of the classical theory of Floquet to provide a unified treatment on the state transformation properties and stability of linear dynamic systems with continuous and piecewise continuous periodic parameters. The linear dynamic system with periodic parameters is represented mathematically by a linear differential equation with periodic coefficients. It is shown that the linear differential equation with periodic coefficients can always be reduced to linear difference equations with constant coefficient matrices. The coefficient matrices of difference equations are all related to each other by similarity transformations. Consequently, it is sufficient to study any one of the difference equations to determine stability of the original differential equation with periodic coefficients. The entire discussion is developed from the point of view of the state transformation which has been proven to be of great use in the study of dynamic systems.

Loss in information transmission through two-way channels

Abstract: A discrete memoryless two-way channel is defined by a set of transmission probabilities {p(y, y¯/x, x¯)}, where x and x¯ are the left and right input signals, and y and y¯ are the left and right received signals, respectively. If the transmission probabilities are restricted so that p(y, y¯/x, x¯) = px¯(y¯/x)p¯x(y/x¯), and if we attach to the two channel terminals independent, finite memory, stationary signal sources which generate channel inputs depending on sequences of past inputs and outputs, then expressions for average information transmission rates in the left-to-right and right-to-left directions can be developed and their sum will be a simple information measure. When mutually independent messages are to be transmitted in opposite directions through the channel, it is desirable that they be encoded into sequences of strategy functions which together with the received signals constitute inputs to a transducer whose outputs are the channel input signals. The message source—encoder—transducer combinations are stochastically equivalent to signal sources whose outputs are governed by appropriate probabilities. We can interpret the transducer—channel combination as a derived two-way channel whose inputs are the strategy functions and whose outputs are the outputs of the underlying channel. Expressions for the information transmission rate through the two directions of the derived channel are developed and are compared to the expressions for the average information about outputs of the equivalent signal sources, transmitted through the underlying two-way channel. The values of the former expressions are found to be less than or equal to the values of the latter, the difference constituting a “coding information loss.” A condition on the transmission probabilities enables us to define a class of lossless channels. Similarly another class is defined having the property that, regardless of the strategy code used, the information transmitted through the derived channel will be strictly less than the information transmitted through the underlying channel. The consequences of the above results on the random selection of message codes are discussed. It is shown that one can obtain the number of variables to be optimized when best random codes for lossy channels are desired, by using the number of variables for lossless channels as an exponent to the product of the size of the input and output signal alphabets. For the lossy channel class a simplified encoding procedure must in practice be applied, but as can be demonstrated, it will not yield optimal codes.

On information rates for finite-state channels

Abstract: This paper presents sufficient conditions for the direct computation of the entropy for functional (nonMarkovian) processes and thus also of the rate of information for finite-state channels. The condition for exponential convergence of the upper and lower bounds on the entropy ( Birch, 1962 ), viz., that the transition probability matrix be strictly positive, is here relaxed. Examples are presented involving simple finite-state channels.

Information pattern, learning structure, and optimal decision rule

Abstract: In Section I various types of information structures in individual decision-making under uncertainty are discussed. A quantity which measures the value of information structure is proposed. In Section II the team decision problem is considered. Finally, in Section III we consider the multistage decision process with learning and we shall try to evaluate the effect of the learning structure and compute it by the technique of dynamic programming.

Stability criteria for a class of nonlinear systems

Abstract: A general class of nonlinear systems is investigated from the stand-point of global asymptotic stability. It is shown that such stability of a unique equilibrium state is guaranteed if the system represented by the first approximation is either unconditionally stable or if the system of the first approximation is conditionally stable and the characteristic values (poles) lie in a limited region of the left-half complex plane. The systems considered are generalizations of the systems proposed by Lur'e and Letov. The stability criteria derived by these authors are shown to be, in general, more restrictive than it is necessary to assure global asymptotic stability. The generalization is achieved by replacing the discrete variables of multivariable autonomous dynamical systems by a parametrized continuum. The introduction of the continuum both broadens the class of systems under consideration and expedites the investigation of the problem of stability.

A note on pattern-recognition techniques and game-playing programs

Abstract: A problem encountered in the design of game-playing programs—the evaluation of board positions—is formally identical to a typical pattern-recognition problem. Certain techniques, which have been found helpful in the pattern-recognition case, may therefore also be applied to game-playing programs. A brief review of these techniques, based on multivariate analysis, is given.