When digital signal processing operations are implemented on a computer or with special-purpose hardware, errors and constraints due to finite word length are unavoidable. The main categories of finite register length effects are errors due to A/D conversion, errors due to roundoffs in the arithmetic, constraints on signal levels imposed by the need to prevent overflow, and quantization of system coefficients. The effects of finite register length on implementations of linear recursive difference equation digital filters, and the fast Fourier transform (FFT), are discussed in some detail. For these algorithms, the differing quantization effects of fixed point, floating point, and block floating point arithmetic are examined and compared. The paper is intended primarily as a tutorial review of a subject which has received considerable attention over the past few years. The groundwork is set through a discussion of the relationship between the binary representation of numbers and truncation or rounding, and a formulation of a statistical model for arithmetic roundoff. The analyses presented here are intended to illustrate techniques of working with particular models. Results of previous work are discussed and summarized when appropriate. Some examples are presented to indicate how the results developed for simple digital filters and the FFT can be applied to the analysis of more complicated systems which use these algorithms as building blocks.

Effects of finite register length in digital filtering and the fast Fourier transform

Models for a class of narrow-band digital filters and a class of wide-band digital filters are developed and analyzed The analysis forms a basis for a computer-aided design technique for such filters. Our aim is to show that useful performance can be achieved within the constraints of very simple realizations.

Designing simple, effective digital filters

Quantitative results are presented concerning the loss in signal detectability in a digital correlator due to quantization, sampling, and correlation. The results and the method for obtaining them are believed to be applicable to a wide class of problems in the design of digital systems and in numerical analysis.

Statistics of the inner product of a quantized, random vector and a constant vector: Effects of component correlation

Exact incremental algorithms are developed for computing the mean and the correlation functions of time signals. Corresponding special-purpose computer organizations for a recursive averager and correlator are suggested.

On Incremental Algorithms ror Averaging and Correlation Computation

An efficient method for calculating the mean values and standard deviations of spectral estimates and decision variables is described which utilizes Kellogg's numerical method for calculating correlation functions of quantized random waveforms. The method and computer program are designed to explore the interaction between sampling (i.e., aliasing) and quantization effects.

http://ieeexplore.ieee.org/iel5/5/31116/01448807.pdf

D.W. Tufts

Papers

Effects of quantization and sampling in digital correlators and in power spectral estimation