A low-density parity-check code is a code specified by a parity-check matrix with the following properties: each column contains a small fixed number j \geq 3 of l's and each row contains a small fixed number k > j of l's. The typical minimum distance of these codes increases linearly with block length for a fixed rate and fixed j . When used with maximum likelihood decoding on a sufficiently quiet binary-input symmetric channel, the typical probability of decoding error decreases exponentially with block length for a fixed rate and fixed j . A simple but nonoptimum decoding scheme operating directly from the channel a posteriori probabilities is described. Both the equipment complexity and the data-handling capacity in bits per second of this decoder increase approximately linearly with block length. For j > 3 and a sufficiently low rate, the probability of error using this decoder on a binary symmetric channel is shown to decrease at least exponentially with a root of the block length. Some experimental results show that the actual probability of decoding error is much smaller than this theoretical bound.

/pdf/low-density-parity-check-codes-vltmv2dex4.pdf

Low-Density Parity-Check Codes

A method is described for constructing long error-correcting codes from one or more shorter error-correcting codes, referred to as subcodes, and a bipartite graph. A graph is shown which specifies carefully chosen subsets of the digits of the new codes that must be codewords in one of the shorter subcodes. Lower bounds to the rate and the minimum distance of the new code are derived in terms of the parameters of the graph and the subeodes. Both the encoders and decoders proposed are shown to take advantage of the code's explicit decomposition into subcodes to decompose and simplify the associated computational processes. Bounds on the performance of two specific decoding algorithms are established, and the asymptotic growth of the complexity of decoding for two types of codes and decoders is analyzed. The proposed decoders are able to make effective use of probabilistic information supplied by the channel receiver, e.g., reliability information, without greatly increasing the number of computations required. It is shown that choosing a transmission order for the digits that is appropriate for the graph and the subcodes can give the code excellent burst-error correction abilities. The construction principles

A recursive approach to low complexity codes

Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k > 2 values (i.e., k "classes"). The definition is acquired by studying collections of training examples of the form (xi, f(xi)). Existing approaches to multiclass learning problems include direct application of multiclass algorithms such as the decision-tree algorithms C4.5 and CART, application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and application of binary concept learning algorithms with distributed output representations. This paper compares these three approaches to a new technique in which error-correcting codes are employed as a distributed output representation. We show that these output representations improve the generalization performance of both C4.5 and backpropagation on a wide range of multiclass learning tasks. We also demonstrate that this approach is robust with respect to changes in the size of the training sample, the assignment of distributed representations to particular classes, and the application of overfitting avoidance techniques such as decision-tree pruning. Finally, we show that--like the other methods--the error-correcting code technique can provide reliable class probability estimates. Taken together, these results demonstrate that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.

/pdf/solving-multiclass-learning-problems-via-error-correcting-372i2wfxkc.pdf

Solving multiclass learning problems via error-correcting output codes

Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k &gt 2 values (i.e., k ``classes''). The definition is acquired by studying collections of training examples of the form [x_i, f (x_i)]. Existing approaches to multiclass learning problems include direct application of multiclass algorithms such as the decision-tree algorithms C4.5 and CART, application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and application of binary concept learning algorithms with distributed output representations. This paper compares these three approaches to a new technique in which error-correcting codes are employed as a distributed output representation. We show that these output representations improve the generalization performance of both C4.5 and backpropagation on a wide range of multiclass learning tasks. We also demonstrate that this approach is robust with respect to changes in the size of the training sample, the assignment of distributed representations to particular classes, and the application of overfitting avoidance techniques such as decision-tree pruning. Finally, we show that---like the other methods---the error-correcting code technique can provide reliable class probability estimates. Taken together, these results demonstrate that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.

Solving Multiclass Learning Problems via Error-Correcting Output Codes

f High-performance, special-purpose computer systems are typically used to meet specific application requirements or to off-load computations that are especially taxing to general-purpose computers. As hardware cost and size continue to drop and processing requirements become well-understood in areas such as signal and image processing, more special-purpose systems are being constructed. However, since most of these systems are built on an ad hoc basis for specific tasks, methodological work in this area is rare. Because the knowledge gaited from individual experiences is neither accumulated nor properly organized, the same errors are repeated. I/O and computation imbalance is a notable example-often, the fact that I/O interfaces cannot keep up with device speed is discovered only after constructing a high-speed, special-purpose device. We intend to help correct this ad hoc approach by providing a general guideline-specifically, the concept of systolic architecture, a general methodology for mapping high-level computations into hardware structures. In a systolic system, data flows from the computer memcory in a rhythmic fashion, passing through many processing elements before it returns to memory, much as blood circulates to and from the heart. The system works like an autombbile assembly line where different people work on the same car at different times and many cars are assembled simultaneously. An assembly line is always linear, however, and systolic systems are sometimes two-dimensional. They can be rectangular, triangular, or hexagonal to make use of higher degrees of parallelism. Moreover, to implement a variety of computations, data flow in a systolic system may be at multiple speeds in multiple directions-both inputs and (partial) results flow, whereas only results flow in classical pipelined systems. Generally speaking, a systolic system is easy to implement because of its regularity and easy to reconfigure (to meet various outside constraints) because of its modularity. The systolic architectural concept was developed at Carnegie-Mellon University,'17 and versions of systolic processors are being designed and built by several industrial and governmental organizations.840 This article reviews the basic principle of systolic architectures and explains why they should result in cost-effective, highperformance special-purpose systems for a wide range of problems.

/pdf/why-systolic-architectures-27b11gsxcr.pdf

Why systolic architectures

Error-correcting codes

In this paper a class of linear error-correcting block codes is investigated. These quasicyclic codes are closely related in structure to cyclic codes although they are not nearly as well understood. It is shown that an important subclass of the class of cyclic codes is almost equivalent to quasicyclic codes. Also, a short computer generated list of the best quasicyclic codes of rate ½ is given. The last result is a proof that there exist very long quasicyclic codes which meet the Gilbert bound.

Some results on quasi-cyclic codes

The channel considered here is a real-number adder. Attention is restricted to the case of two sources using binary signals; hence the channel has a ternary output. The paper treats both the noiseless and noisy situations. For the noiseless case the following results are obtained: several code-construction techniques are presented; a Gilbert-like lower bound is derived; an upper bound which shows linear codes to be markedly inferior to nonlinear codes on this channel is derived. It has previously been shown that, subject to certain constraints, the capacity of this channel using multiple-access signaling is 50% greater than the capacity if the users time-share the channel. The present results show that about half of this increase is achievable with simply implemented coding techniques. For the noisy adder channel we obtain the following results: partial upper bounds on the achievable rate region are derived; codes are constructed by concatenating single-user codes and the codes are constructed for the noiseless channel; the minimum distance of these codes is lower bounded; three decoding procedures are analyzed. These results show how independent additive errors affect the code rates achievable on this channel.

Coding for a multiple-access channel*

The BCH bound approaches zero with increasing code length for any fixed value of the ratio of minimum distance to code length. Does this happen because the codes actually deteriorate or does the bound just become loose? It is shown that the BCH bound gives the true minimum distance within a factor of two. We conclude that it is the codes, not the bound, which deteriorate with increasing code length.

Long BCH Codes Are Bad

The notion of an extended cyclic code has recently been formulated. It is known that, subject to certain polynomial constraints, certain random-error-correcting cyclic codes can be extended to form codes capable of automatically recovering from synchronization losses. It is shown in this paper that the above-mentioned constraints are unnecessary and that this synchronization recovery technique can be used with any additive-error-correcting cyclic code.

E. J. Weldon

Papers

Error-correcting codes

Some results on quasi-cyclic codes

Coding for a multiple-access channel*

Long BCH Codes Are Bad

A note on synchronization recovery with extended cyclic codes