Showing papers in "Information & Computation in 1960"

On a class of error correcting binary group codes

Abstract: A general method of constructing error correcting binary group codes is obtained. A binary group code with n places, k of which are information places is called an (n,k) code. An explicit method of constructing t-error correcting (n,k) codes is given for n = 2m−1 and k = 2m−1−R(m,t) ≧ 2m−1−mt where R(m,t) is a function of m and t which cannot exceed mt. An example is worked out to illustrate the method of construction.

Further results on error correcting binary group codes

Abstract: The present paper is a sequel to the paper “On a class of error-correcting binary group codes”, by R. C. Base and D. K. Ray-Chaudhuri, appearing in Information and Control in which an explicit method of constructing a t-error correcting binary group code with n = 2m − 1 places and k = 2m − 1 − R(m,t) ≧ 2m − 1 − mt information places is given. The present paper generalizes the methods of the earlier paper and gives a method of constructing a t-error correcting code with n places for any arbitrary n and k = n − R(m,t) ≧ [(2m − 1)/c] − mt information places where m is the least integer such that cn = 2m − 1 for some integer c. A second method of constructing t-error correcting codes for n places when n is not of the form 2m − 1 is also given.

Some further notes on a class of skew distribution functions

Abstract: This note takes issue with a recent criticism by Dr. B. Mandelbrot of a certain stochastic model to explain word-frequency data. Dr. Mandelbrot's principal empirical and mathematical objections to the model are shown to be unfounded. A central question is whether the basic parameter of the distributions is larger or smaller than unity. The empirical data show it is almost always very close to unity, sometimes slightly larger, sometimes smaller. Simple stochastic models can be constructed for either case, and give a special status, as a limiting case, to instances where the parameter is unity. More generally, the empirical data can be explained by two types of stochastic models as well as by models assuming efficient information coding. The three types of models are briefly characterized and compared.

A program for correcting spelling errors

Abstract: A program using a simple, heuristic procedure for associating “similar” spellings is able to correct misspelled words. Given only a vocabulary of properly spelled words, the computer can correct most (including unanticipated) misspellings without human assistance. Apart from practical applications, the process is interesting as an example of an unusual form of pattern recognition.

Two-error correcting Bose-Chaudhuri codes are quasi-perfect

Abstract: It is shown that all two-error correcting Bose-Chaudhuri codes are close-packed and therefore optimum. A method is also given for finding cosets of large weight in t > 2-error correcting Bose-Chaudhuri codes, which suggests that no other nontrivial codes are close-packed.

A generalization of the sampling theorem

Abstract: The sampling theorem for bandlimited functions allows one to reconstruct exactly a function containing no frequencies higher than W cps, given the values of the function at equispaced sampling points ( R + 1)/ W sec apart. This theorem is generalized to allow reconstruction, given the values of the function and its first R derivatives at equispaced sampling points, ( R + 1)/2 W sec apart. For large R , the R -derivative expansion approaches a Taylor's series weighted by a Gaussian density about each sample point.

Symbolic Analysis of a Decomposition of Information Processing Machines

Abstract: In this paper we study the problem of replacing (decomposing) a complex finite state sequential machine by several simpler ones which operate in parallel and yield the same result. In the first part, we give the necessary mathematical background and results. In the second part, we apply these results and derive the necessary and sufficient conditions for the existence of a decomposition for a given machine. If a decomposition exists then the required simpler machines which have to be connected in parallel are given.

Minimum coefficient rate for stationary random processes

Abstract: Let χ1(t), χ2(t), …, χN(t) be N sample functions of a stationary random process with mean zero, variance one, and spectral density function S(f). For large N and T it is possible to construct a good approximation to the product χ1(t1)χ2(t2) … χN (tN) on the N-dimensional cube 0 ≦ ti ≦ T by using approximately TN exp[—N ∫∞—∞ S(f) log S(f) df] terms of an infinite series. This can be interpreted as saying that one needs exp[—∫∞—∞S(f) log S(f) df] numbers per function per unit time to determine the product. Some results are also obtained on the construction of approximations to the individual functions χk (t) from the approximate product. If S(f) has a constant value in the band (−W, W) and vanishes outside, this approach yields the well-known result that 2W numbers per function per unit time are required.

A note on human recognition of hand-printed characters

Abstract: Nine human observers were given the task of identifying isolated hand-printed characters. Their individual accuracies ranged from 94.9% to 96.5%, and even their pooled best guess was right only 96.8% of the time. These figures can serve as standards for the accuracy of mechanical devices for letter-recognition.

The theoretical channel capacity of a single neuron as determined by various coding systems

Abstract: The information channel capacity of a model neuron with a fixed refractory period δ has been calculated for optimum continuous time interval coding. Two types of noise perturbations have been considered, a Gaussian probability distribution and a rectangular distribution of the time of occurrence of the response to a stimulus. Laboratory measurements indicate a Gaussian distribution with a standard deviation of latency σ, of about 5 μsec gives the best fit to an actual nerve fiber. This results in a maximum information transmission of slightly over 4000 bits per second. The results are compared with the discrete theory of MacKay and McCulloch.

Note on the boolean properties of context free languages

Abstract: A language is considered to be any set of sentences (strings) made up from a finite vocabulary (alphabet). A grammar is a device which enumerates a language, e.g., a Turing machine with any set of input signals, a finite automaton, or a Post system. One often considers the family of languages which have grammars meeting certain specifications or restrictions, e.g., the finite state languages (the languages generatable by finite automata) . A major question of interest is whether such a family is a closed system with respect to the Boolean operations: set-theoretic union, intersection, and difference. The family of recursively enumerable languages is well known to be closed under union and intersection, but not under difference. I t is also well known that the family of finite state languages is closed under all three operations. In this note we answer this question for another family of languages, called "type 2" or "context free" by Chomsky (1959). For the sake of having a convenient reference we shall present in modified form Chomsky's definitions. I t will be seen that a context free grammar is essentially a special case of a semi-Thue system, cf. Davis (1958). We consider a context free (CF) grammar to be a finite set G of "rewriting rules" a --~ ~, where a is a single symbol and ~ is a finite string of symbols from a finite alphabet (vocabulary) V. V contains precisely the symbols appearing in these rules plus the "boundary" symbol ~, which does not appear in these rules. Rules of the form a --~ a (which

A note on the application of graph theory to digital computer programming

Abstract: A graph-theoretic model for the description of flowcharts and programs is defined. It is shown that properties of directed graphs and the associated connection matrices can be used to detect errors and eliminate redundancies in programs. These properties are also used in the synthesis of composite programs. Finally, the model is expanded to take into account frequencies of execution of portions of a program, and a problem concerning optimum arrangement of a program in storage is solved.

A statistical model for interpreting neuroelectric responses

Abstract: Neuroelectric activity recorded after presentation of a controlled stimulus is called an “evoked response.” Evoked responses are random, in that repeated presentations of a stimulus do not produce identical responses despite all effort to maintain identical conditions for all stimulus presentations. Responses may then be described statistically. The present model attempts to relate statistical characteristics of evoked responses, recorded by gross electrodes, to the statistical activity of the neural elements that contribute to the responses. The model postulates one or more populations of elements which, when they fire, contribute elemental waveforms to a gross response in which these waveforms are linearly summed. The statistical behavior of the elements in a population is described by their instantaneous firing rate, which is a function of time. In terms of the model, the statistical properties of the gross response (such as the mean and variance, both of which are functions of time) are shown to be related in a simple way to the instantaneous firing rates and elemental waveforms of the populations that contribute to the response. The model is an extension of the shot-noise model to time-variant phenomena; but some of the assumptions of the shot-noise model (specifically, statistical independence of firings) are relaxed.

The extended entropy uncertainty principle

Abstract: A minimum principle is obtained for the entropy in the Wigner-Moyal distribution of linear position and momentum. This result supports the conjecture, previously discussed in the statistically independent case, that the joint dimensionless entropy of noncommuting observables has a positive minimum. The minimizing wave functions are Gaussian with complex-valued variances.

On the efficiency of a new method of dictionary construction

Abstract: Programs for constructing dictionaries of texts, with computers, have sometimes been adaptat ions of methods suitable for manual construction with card indexes. With all card index methods it is customary to keep the different words collected in alphabetical order, for the structure of a card index lends itself to such a process: all tha t is necessary to insert ~ new word into the index between two existing ones is to make out a new card and to put it in the correct place. However, the insertion of a new word in the store of a computer where the words are kept in alphabetical order is a t ime-consuming process, for all the words below the one which is inserted have to be \"moved down\" by one place. If, however, a computer method is used where the words are not stored in alphabetical order but in the order in which they occur, the position is even worse, for although the shifting of words is eliminated, the dict ionary search which is necessary for each word in the text, to establish whether it is a new word or not, must involve all the words collected so far, and not: iust a small number of them, as would be the ease if the words were in alphabetical order, and a logarithmic search (Booth, 1955) could be used. The method of construction described in this paper overcomes both these problems. I t is no t an adaptat ion of a manual method, but is designed specifically for computers. The method is based on a tree structure, which is discussed below.

The redundancy of texts in three languages

Abstract: The procedure that predicts the mean information per letter in a long text by adding the constraint measured between pairs of letters in a text has been tested more fully. Results are presented to show that with randomized texts there is a close approximation to the Miller-Madow prediction of sample bias. Three samples of English of varying complexity show slightly more information per single letter and much more information in an average letter for the more difficult material. Conversely, samples from Samoan, English, and Russian show some constancy in the average information per letter in spite of wide differences in size of their alphabets. Thus, greater redundancy is correlated with a larger alphabet.

On control of linear systems with time lags

Abstract: The terminal control of a general time invariant linear system with time lags has been studied in a variety of situations by means of the technique of dynamic programming. Optimum control laws, satisfying a minimum energy constraint are explicitly derived under each of the following assumptions concerning the noise in the system: noise free, noise generated by a Markov source, noise whose statistics are partially unknown, and worst case or competitive disturbances.

Loss and recovery of information by coarse observation of stochastic chain

Abstract: In a stationary stochastic chain, the states are grouped into classes or coarsely defined macrostates, engendering another chain defined in terms of macrostates. Comparison of information content of these two chains is made in detail. Loss of information caused by the coarseness of the definition of macrostates can be recovered, partially or totally depending on the case, when there is correlation in the chain. The range of correlation in some cases is increased by the coarse definition of states, thus creating a longer “aftereffect.” If the correlation is weak, this aftereffect tapers off exponentially with time.

A class of definitions of “duration” (or “uncertainty”) and the associated uncertainty relations

Abstract: A new class of definitions for “time duration” and “bandwidth” (or “time uncertainty” and “frequency uncertainty”), in terms of norms of L p spaces, is suggested. Some properties of the definitions and the associated uncertainty relations are derived. As examples of the application of these concepts, the problems of the approach of the probability distribution of shot noise towards the normal law, and the “beamwidth”—“aperture width” product in antenna theory are considered.

A note on the strong converse of the coding theorem for the general discrete finite-memory channel

Abstract: The strong converse for the discrete memoryless channel was proved by the author (Wolfowitz, 1957). (The result actually proved is stronger than the strong converse because of the O(~¢/n) term in the exponent). Subsequently the author (1958) and Feinstein (1959) independently gave the capacity C of a discrete finite-memory channel, and proved the strong converse of the coding theorem for the special discrete finitememory channel studied (Wolfowit~, 1957, 1958). In the present note we prove the strong converse for the general discrete finite-memory channel. Thus our result includes t ha t of Wolfowitz (1958) and Feinstein (1959) as a special case. The proof is a slight modification of the proof of Wolfowitz (1958), whose notation and definitions are hereby assumed. For a definition of the capacity C see (Wolfowitz, 1958) or (Feinstein, 1959); for a definition of the general discrete finite-memory channel see (Feinstein, 1959) or (Feinstein, 1958, p. 90). We shall assume without essential loss of generality that both the input and output alphabets consist of two letters, say 0 and l; extension to the case where each alphabet contains any finite number of symbols is trivial. Any sequence of n zeros or ones will be called an n-sequence. A code (N, X) is a set

Effective sampling rates for signal detection; or can the Gaussian model be salvaged?

Abstract: The usual justification for talking about signal-to-noise ratio is in terms of a Gaussian model. This model can be treated either by means of information theory or by means of (earlier) methods of statistical inference. In either case the justification is often achieved by assuming that sampling is done at the Nyquist rate. This justification collapses if we are given a record of finite duration, since the sampling theorem is then inapplicable. In fact the Gaussian model itself collapses since it leads to the absurd conclusion that an infinite amount of information can be obtained in a finite time. But the mathematical convenience of the Gaussian model cannot be lightly brushed aside. The intention of this paper is primarily to try to salvage the Gaussian model by assuming that there is an effective sampling rate that cannot be exceeded. This rate could be slower or faster than the Nyquist rate. If an inefficient, but pleasantly simple statistic (the “power statistic”) is used, then there is less point in sampling faster than the Nyquist rate. For the reader's convenience, some material on spectral analysis and other matters is collected together in the Appendices. Most of it could be found, explicitly or implicitly, in previous literature. The notion of interaction for weight of evidence, and its relationship to spectral analysis, is explained in Appendix 6, and does not seem to have been previously published.

Paradoxes related to the rate of transmission of information

Abstract: This paper discusses paradoxes related to the possibility of infinite information capacity of certain types of channels. First, a paradox of this type is derived which shows that such paradoxes are not necessarily dependent on the assumption of Gaussian statistics. Next, in the case where signal and noise are assumed to be Gaussian, a different example of this type of paradox is derived; also, a necessary and sufficient condition for the avoidance of this form of the paradox is derived. This condition is shown to be satisfied in a class of plausible physical situations.

The application of the Ferranti Mercury computer to linguistic problems

Abstract: In a book published in 19581 (Booth et al., 1958) and hereafter called MRLP Booth, Brandwood, and Cleave discussed at some length the various ways in which electronic computers could be applied to resolving linguistic problems, and gave an account of the results obtained at the Birkbeck College Computational Laboratory. Since that time, the University of London has acquired a Ferranti Mercury computer which, though primarily designed for the solution of mathematical problems (and making use of floating point arithmetic), has a large immediate-access store and a great speed which make it very suitable for linguistic work. This acquisition has made it possible to put into practice on a full scale many of the ideas which the book contained. The present paper elaborates upon some of these ideas with particular reference to the way in which they are affected by the design of the Mercury. The paper is divided into several sections and the first devoted to a description of the Mercury insofar as it affects the design of linguistic programmes. The basic ideas behind the dictionary searching methods described in Section V appeared in MRLP. The minor discrepancies between some of the mathematical results in this section and the corresponding ones in MRLP are due to the assumption in the latter that this size of dictionary is large. It will become apparent that this assumption cannot always be made in dealing with the Mercury.

The automatic construction of a glossary

Abstract: In this paper the term “glossary” refers to a list, in alphabetical order, of all the different words used in any text, together with a statement of how many times each word occurs in the text. The manual construction of a glossary is usually made with the aid of a card index, where one card is used to note the occurrences of each different word in the text. The procedure consists of attempts to look up each consecutive word of the text in the card index; if the word is found, its present occurrence is noted on the card; but if it is not found, a new card is made out and inserted into the index. The automatic construction of a glossary is similar in principle, but the logical rules by which the construction is made are more complicated because the store of an electronic computer, unlike a card index, is limited in size and, in general, is not large enough to contain the entire glossary of a text of average length. There are many ways of overcoming this difficulty and two of the more reasonable ones are discussed below.