Showing papers in "IEEE Transactions on Information Theory in 1977"
TL;DR: The compression ratio achieved by the proposed universal code uniformly approaches the lower bounds on the compression ratios attainable by block-to-variable codes and variable- to-block codes designed to match a completely specified source.
Abstract: A universal algorithm for sequential data compression is presented. Its performance is investigated with respect to a nonprobabilistic model of constrained sources. The compression ratio achieved by the proposed universal code uniformly approaches the lower bounds on the compression ratios attainable by block-to-variable codes and variable-to-block codes designed to match a completely specified source.
5,844 citations
TL;DR: A new multilevel coding method that uses several error-correcting codes that makes effective use of soft-decisions to improve the performance of decoding and is superior to other multileVEL coding systems.
Abstract: A new multilevel coding method that uses several error-correcting codes is proposed. The transmission symbols are constructed by combining symbols of codewords of these codes. Usually, these codes are binary error-correcting codes and have different error-correcting capabilities. For various channels, efficient systems can be obtained by choosing these codes appropriately. Encoding and decoding procedures for this method are relatively simple compared with those of other multilevel coding methods. In addition, this method makes effective use of soft-decisions to improve the performance of decoding. The decoding error probability is analyzed for multiphase modulation, and numerical comparisons to other multilevel coding systems are made. When equally complex systems are compared, the new system is superior to other multilevel coding systems.
1,070 citations
TL;DR: With the Delsarte-MacWilliams inequalities as a starting point, an upper bound is obtained on the rate of a binary code as a function of its minimum distance, which is asymptotically less than Levenshtein's bound and so also Elias's.
Abstract: With the Delsarte-MacWilliams inequalities as a starting point, an upper bound is obtained on the rate of a binary code as a function of its minimum distance. This upper bound is asymptotically less than Levenshtein's bound, and so also Elias's.
473 citations
TL;DR: The coding theorem and strong converse for this communication situation is proved for the case when one of the private messages has rate zero.
Abstract: A broadcast channel with one sender and two receivers is considered. Three independent messages are to be transmitted over this channel: one common message which is meant for both receivers, and one private message for each of them. The coding theorem and strong converse for this communication situation is proved for the case when one of the private messages has rate zero.
356 citations
TL;DR: The reduced update Kalman filter is shown to be optimum in that it minimizes the post update mean-square error (mse) under the constraint of updating only the nearby previously processed neighbors.
Abstract: The Kalman filtering method is extended to two dimensions. The resulting computational load is found to be excessive. Two new approximations are then introduced. One, called the strip processor, updates a line segment at a time; the other, called the reduced update Kalman filter, is a scalar processor. The reduced update Kalman filter is shown to be optimum in that it minimizes the post update mean-square error (mse) under the constraint of updating only the nearby previously processed neighbors. The resulting filter is a general two-dimensional recursive filter.
347 citations
TL;DR: The advances in the area of multi-way channels during the period 1961-1976 are described in this paper, where Shannon's two-way channel, the multiple access channel, interference channel, broadcast channel, and relay channel are treated successively.
Abstract: The advances in the area of multi-way channels during the period 1961-1976 are described Shannon's two-way channel, the multiple-access channel, the interference channel, the broadcast channel, and the relay channel are treated successively Only channel coding aspects are discussed Thirty-nine open problems are mentioned
332 citations
TL;DR: Degraded two-user channels are introduced and studied in detail; in particular, an achievable region is obtained by combining two regions that correspond to the two different modes of transmission.
Abstract: A discrete memoryless channel with two inputs and two outputs, called a two-user channel, is studied under the communication situation where only separate messages are allowed to be sent between two source-user pairs. An outer bound to the capacity region is obtained by a method similar to that used by the author for the broadcast channel. Two extreme cases of two-user channels are discussed: separate channels and incompatible channels. Degraded two-user channels are introduced and studied in detail; in particular, an achievable region is obtained by combining two regions that correspond to the two different modes of transmission. This idea is extended to the general two-user channel where an achievable region for the general channel is obtained by random coding arguments.
240 citations
TL;DR: It is concluded that the requirement of skewsymmetry constitutes a powerful and effective sieve for the search for near-optimal sequences.
Abstract: The value 2e^{2} is calculated by approximations as a conjectured asymptotic limit for the best central to sidelobe energy ratio of very long low autocorrelation binary sequences. The same bound is calculated by approximation for skewsymmetric sequences, hence it is concluded that the requirement of skewsymmetry constitutes a powerful and effective sieve for the search for near-optimal sequences. A second sieve based on the use of complementary sequences is discussed and shown to be quite effective for sequence lengths up to 31, and its investigation for longer lengths by means of computers is suggested. A third and a fourth sieve, based on the selection of restricted classes of complementary sequences, are also discussed.
228 citations
TL;DR: Optimality is proved by showing that every balanced binary sequence has at least two distinct out-of-phase correlation values which are at least as large as those obtained here.
Abstract: The construction of a class of balanced binary sequences with optimal autocorrelation properties is described. Given any odd prime p and any positive integer m , a balanced ( \pm 1) binary sequence of length p^{m} - 1 whose cyclic autocorrelation function c (\tau) satisfies c (0) = p^{m} - 1 , and, for \tau
eq 0, c (\tau) = +2 or -2 when (p^{m} - 1)/2 is odd, and c(\tau) = 0 or -4 when (p^{m} - 1)/2 is even is constructed. Optimality is proved by showing that every balanced binary sequence has at least two distinct out-of-phase correlation values which are at least as large as those obtained here.
180 citations
TL;DR: Tree encoding and sequential decoding are considered for noisy channels that respond a random number of times to each input in mathematical models of certain speech recognition systems.
Abstract: Tree encoding and sequential decoding are considered for noisy channels that respond a random number of times to each input. Such channels appear in mathematical models of certain speech recognition systems. The decoding error probability and the channel capacity are bounded by extension of the methods of Jelinek and Zigangirov to noisy multilevel channels with input-dependent insertions. Certain analytical difficulties peculiar to the channels in question are indicated.
179 citations
TL;DR: The equivalence of M -sequence matrices to Walsh-Hadamard matrices can be exploited to take advantage of the latter's fast transform algorithm.
Abstract: The equivalence of M -sequence matrices to Walsh-Hadamard matrices can be exploited to take advantage of the latter's fast transform algorithm. The nature of the obvious equivalence, its relation to the M -sequence and its implementation as address modifications realized by linear feedback shift-registers are discussed.
TL;DR: Shannon's information-theoretic approach to cryptography is reviewed and extended and it is shown that Shannon's random cipher model is conservative in that a randomly chosen cipher is essentially the worst possible.
Abstract: Shannon's information-theoretic approach to cryptography is reviewed and extended. It is shown that Shannon's random cipher model is conservative in that a randomly chosen cipher is essentially the worst possible. This is in contrast with error-correcting codes where a randomly chosen code is essentially the best possible. The concepts of matching a cipher to a language and of the trade-off between local and global uncertainty are also developed.
TL;DR: A generalized Gram-Charlier series, applicable to non-Gaussian problems, is developed and the high inherent accuracy of the series is demonstrated by development of the expansion for the sum of independent, identically distributed log-normal variates.
Abstract: A generalized Gram-Charlier series, applicable to non-Gaussian problems, is developed. Expressions are given for the first six error coefficients. The high inherent accuracy of the series is demonstrated by development of the expansion for the sum of independent, identically distributed log-normal variates.
TL;DR: This volume contains advanced materials of queueing theory plus three application chapters, and gives a comprehensive account of the diffusion process approximation and heavy traffic theory as applied to queueing systems.
Abstract: Chapter 1, \" Queueing systems, \" is an introductory chapter. Chapter 2, \" Some important random processes, \" reviews those random processes pertinent to queueing systems, e.g., Markov chains, Markov processes, and birth-death processes. Chapter 3, \" Birth-death queueing systems in equilibrium, \" deals with a class of queueing systems with Poisson arrivals (possibly with queue-dependent rate) and exponential service times. In Chapter 4, \" Markovian queues in equilibrium, \" the author discusses queueing systems with Erlangian service time or interarrival time, bulk arrival systems, and the like. Networks of queues with exponential servers are discussed in the last section. Chapter 5, \" The queues M/G/I, \" and Chapter 6, \" The queue GIMIm, \" are treated using the imbedded Markov chain technique. Most of the material presented in Chapters 2 through 6 are found in many previously published books, but Kleinrock's treatment is perhaps the most comprehensive. Chapter 7, \" The method of collective marks, \" is a unique chapter. The z-transform and Laplace-transform, originally introduced in Chapter 2 as formal solution techniques, are now given a physical and probabilistic interpretation. The last chapter of Vol. I is Chapter 8, \" The queue G/G/l, \" which deals with Lindley's theory for solving the waiting time distribution in the G/G/l queue. The mathematical problem is reduced to an integral equation of Wiener-Hopf type and its corresponding spectral factorization problem. The reader of this TRANSACTIONS will recognize an analogy' between Lindley's theory and Wiener's filtering theory. VOLUME II This volume contains advanced materials of queueing theory plus three application chapters. Chapter 1, \" A queueing theory primer, \" is a 22-page summary of Volume I. Chapter 2, \" Bounds, inequalities and approximations, \" is perhaps the most novel and worthwhile chapter in this two-volume series. Exponentially tight bounds on the tail of the waiting time distribution are discussed with much the same spirit as the bounding arguments that the information theorist applies to the probability of decoding errors: the Chernoff bound finds a beautiful application in queueing theory also! The author also gives a comprehensive account of the diffusion process approximation and heavy traffic theory as applied to queueing systems. Chapter 3, \" Priority queue, \" is also a unique and valuable chapter. Much of the material is based on the author's earlier work: conservation laws, time-dependent priorities, and optimal bribing for queue position. Chapter 4, \" Computer time-sharing and multiaccess …
TL;DR: A technique is presented to determine the region of achievable rates for some source and channel networks and this technique is applied to the solution of a source:network problem.
Abstract: A technique is presented to determine the region of achievable rates for some source and channel networks. This technique is applied to the solution of a source:network problem that seems to be the simplest illustration of a new typical difficulty in coding for source networks: namely, when the same encoding of a source is required to meet the conflicting demands of 1) supplying side-information to the decoder of another source, and 2) providing direct-information to its own decoder in company with other side-information.
TL;DR: A source coding problem is considered which generalizes source coding with side information and the resuit is generalized to an arbitrary finite number of decoders.
Abstract: A source coding problem is considered which generalizes source coding with side information [1], [2]. Three correlated information sources X,Y and Z , are block-encoded: Y is to be reconstructed by two different decoders, one having access to the encoded version of X and the other having access to the encoded version of Z . The region of achievable rates is determined, assuming that thc sources are discrete, memoryless, and stationary. The resuit is generalized to an arbitrary finite number of decoders.
TL;DR: This constrained optimization problem is reduced to an unconstrained dual problem with a nondifferentiable cost and then solved using a steepest ascent algorithm based on directional differentials to provide a versatile alternative to the least-squares methods that are commonly applied to signal processing problems.
Abstract: In the envelope-constrained filtering problem, one attempts to optimize a filter's impulse response subject to the constraint that its response to a specified input lies within a prescribed envelope. This constrained optimization problem is reduced to an unconstrained dual problem with a nondifferentiable cost and then solved using a steepest ascent algorithm based on directional differentials. The resulting technique provides a versatile alternative to the least-squares methods that are commonly applied to such problems. Applications to a wide variety of signal processing problems in areas such as communication channel equalization, television transmission, radar and sonar detection, filter and antenna design, seismology, and ultrasonic imaging are discussed, and some numerical results are presented.
TL;DR: It is shown that it is possible to derive an algorithm which on-line finds the optimal fixed-lag smoother, a self-tuning smoother, which has good transient, as well as good asymptotic, properties.
Abstract: The problem of estimating a discrete-time stochastic signal which is corrupted by additive white measurement noise is discussed. How the stationary solution to the fixed-lag smoothing problem can be obtained is shown. The first step is to construct an innovation model for the process. It is then shown how the fixed-lag smoother can be determined from the polynomials in the transfer function of the innovation model. In many applications, the signal model and the characteristics of the noise process are unknown. It is shown that it is possible to derive an algorithm which on-line finds the optimal fixed-lag smoother, a self-tuning smoother. The self-tuning smoother consists of two parts: an on-line estimation of the parameters in the one-step ahead predictor of the measured signal, and a computation of the smoother coefficients by simple manipulation of the predictor parameters. The smoother has good transient, as well as good asymptotic, properties.
TL;DR: It is shown how to modify RSD so that no more than \psi states are retained and how to assess the degradation of the error exponent.
Abstract: In the detection of m -level pulse-amplitude modulation (PAM) signals transmitted over a noisy linear
u -symbol memory channel, maximum likelihood sequence detection (MLSD) is asymptotically optimum in the sense that the exponent of the probability of symbol error in the limit of small noise is the largest that can be achieved by any detector. A reduced state detection (RSD) is developed with the aim of mitigating the computational complexity of MLSD while attaining the same error exponent as MLSD. In RSD, for each baud, the recent received signal levels delineate a certain list of the most probable of the m^{nu} possible states. Retention of only the paths threading through these most probable states is the key to success of RSD. Suppose it is required that the number of states retained at any time not exceed \psi . When \psi is less than the number of states required for RSD, it is shown how to modify RSD so that no more than \psi states are retained and how to assess the degradation of the error exponent.
TL;DR: The stability and convergence of the fast start-up algorithm with "stochastic" update in the presence of noise are examined and the one-shot method, which involves estimation of the inverse filter from one period of the received signal, proves to be robust in the absence of noise.
Abstract: Prior knowledge of the typical signal spectrum at the equalizer input can be used to speed up the convergence of a least mean-square error nonrecursivc equalizer by placing a weighting matrix in the path of the coefficient (tap gain) corrections. When the training sequence is periodic with period equal to the time-spread of the nonrecursive equalizer, the weighting matrix is not only symmetric and Toeplitz, but also circulant. Thus the fast update algorithm can be implemented by inserting a single nonrecursive filter in the path of the periodic input sequence before it is used for computing coefficient correction terms. The stability and convergence of the fast start-up algorithm with "stochastic" update in the presence of noise are examined. In the absence of prior information about the channel characteristics, this fast converging training procedure may be easily extended to a one-shot method of equalizer setup. The one-shot method, which involves estimation of the inverse filter from one period of the received signal, proves to be robust in the presence of noise. Simulation resuits are given for class I partial-response and Nyquist (20 percent roll-off raised cosine) data transmission systems with quadrature amplitude modulation over a dial-up telephone connection.
TL;DR: It is shown for a certain class of sources that the problem of designing good trellis codes is equivalent to that of simulating general random processes by filtering digital memoryless sources.
Abstract: The theory of sliding-block codes (nonlinear, time-invariant, discrete-time filters) is employed to obtain general source coding theorems for ergodic sources using time-invariant trellis coding (time-invariant decoding filter and replicating trellis). The results are coupled with the theory of universal block source codes to obtain universal trellis source coding theorems for classes of sources. It is shown for a certain class of sources that the problem of designing good trellis codes is equivalent to that of simulating general random processes by filtering digital memoryless sources.
IBM1
TL;DR: A correspondence between linear (n,k,d) codes and algorithms for computing a system of k bilinear forms is established, holding promise of a better understanding of the structure of existing codes as well as for methods of constructing new codes with prescribed rate and distance.
Abstract: A correspondence between linear (n,k,d) codes and algorithms for computing a system \psi of k bilinear forms is established under which the codelength n is equal to the multiplicative complexity of the algorithm for computing \psi , and the code distance d is underbounded by the minimum number of multiplications required to compute any linear combination of the k forms in \psi . This hitherto unexplored approach to linear codes holds promise of a better understanding of the structure of existing codes as well as for methods of constructing new codes with prescribed rate and distance.
TL;DR: It is shown that it is possible to send at capacity on the main link and still keep the wiretapper's information equal to zero on many, large, arbitrary portions of the message.
Abstract: Wyner recently introduced the concept of a wiretap channel and showed that by transmitting at a rate less than capacity on the main link it was possible to keep the wiretapper's information about the entire message equal to zero. It is shown that it is possible to send at capacity on the main link and still keep the wiretapper's information equal to zero on many, large, arbitrary portions of the message.
TL;DR: It is shown that, for a Gaussian process and for some non-Gaussian processes, any memoryless nonlinearity has a whitening effect in the sense that the output spectrum is smoother and occupies a greater bandwidth than the input spectrum.
Abstract: It is shown that, for a Gaussian process and for some non-Gaussian processes, any memoryless nonlinearity has a whitening effect in the sense that the output spectrum is smoother and occupies a greater bandwidth than the input spectrum. The ratio between input and output bandwidths is investigated by using several measures of bandwidth. Also, it is shown that classes of nonlinearities exist that are equivalent in the sense of producing the same spectral transformations.
TL;DR: A general approach to the design of receivers that are asymptotically most robust is established and a detector is derived by applying the procedure to the special case obtained when the noise density family is defined.
Abstract: The problem of detection of known signals in additive noise is solved under the assumption that the unknown noise density is a member of some known family of symmetric densities. A general approach to the design of receivers that are asymptotically most robust is established. As an example, a detector is derived by applying the procedure to the special case obtained when the noise density family is defined by F=\left\{f \left| \int^{a}_{-a} f(x) dx = p, f \mbox{symmetric} \right}. Simulation results showing the detector's performance for small sample sizes are provided.
TL;DR: Rate-distortion functions for 2-dimensional homogeneous isotropic images are compared with the performance of five source encoders designed for such images and 6-pel DPCM with entropy coding performed best with the mean-square error distortion measure.
Abstract: Rate-distortion functions for 2-dimensional homogeneous isotropic images are compared with the performance of five source encoders designed for such images. Both unweighted and frequency weighted mean-square error distortion measures are considered. The coders considered are a) differential pulse code modulation (DPCM) using six previous samples or picture elements (pels) in the prediction--herein called 6-pel DPCM, b) simple DPCM using single-sample prediction, c) 6-pel DPCM followed by entropy coding, d) 8 \times 8 discrete cosine transform coding, and e) 4 \times 4 Hadamard transform coding. Other transform coders were studied and found to have about the same performance as the two transform coders above. With the mean-square error distortion measure, 6-pel DPCM with entropy coding performed best. Next best was the 8 \times 8 discrete cosine transform coder and the 6-pel DPCM--these two had approximately the same distortion. Next were the 4 \times 4 Hadamard and simple DPCM, in that order. The relative performance of the coders changed slightly when the distortion measure was frequency weighted mean-square error. From R = 1 to 3 bits/pel, which was the range studied here, the performances of all the coders were separated by only about 4 dB.
TL;DR: The maximum entropy method provides an estimate of the power spectral density which maximizes the entropy of a stationary random process from the first N lags of the autocorrelation function.
Abstract: The maximum entropy method provides an estimate of the power spectral density which maximizes the entropy of a stationary random process from the first N lags of the autocorrelation function The method is applied to cases where some of the first N autocorrelation function lags are unknown
TL;DR: It is shown that for "symmetric" wiretap channels, \Gamma(R) is equal to the difference between the capacities of the main channel and the channel from the sender to the wiretapper.
Abstract: The special class of wiretap channels for which a certain quantity \Gamma(R) is constant is examined, and some useful characterizations of its members are given. It is shown that for "symmetric" wiretap channels, \Gamma(R) is equal to the difference between the capacities of the main channel and the channel from the sender to the wiretapper.
TL;DR: New analytical results are provided which permit significant reduction in the amount of computation needed to evaluate autocorrelation and cross-correlation parameters which characterize the performance of asynchronous phase-coded spread-spectrum multiple-access communication systems.
Abstract: The selection of sets of periodic sequences with good correlation parameters is an important problem in many areas of communication theory. Although algorithms have been proposed for selecting sets of sequences, the amount of computation required is often prohibitive. Several autocorrelation and cross-correlation parameters are investigated such as those which characterize the performance of asynchronous phase-coded spread-spectrum multiple-access communication systems. New analytical results are provided which permit significant reduction in the amount of computation needed to evaluate these correlation parameters.