Adaptive channel memory truncation for maximum likelihood sequence estimation

TLDR: This work proposes a linear prefilter to force the overall impulse response of the channel/prefilter combination to approximate a desired truncated impulse response (DIR) of acceptably short duration and shows that the minimum mean-square error can be expressed as the minimum eigenvalue of a certain channel-dependent matrix, and that the corresponding eigenvector represents the optimum DIR.

Abstract: Maximum likelihood data sequence estimation, implemented by a dynamic programming algorithm known as the Viterbi algorithm (VA), is of considerable interest for data transmission in the presence of severe intersymbol interference and additive Gaussian noise. Unfortunately, the required number of receiver operations per data symbol is an exponential function of the duration of the channel impulse response, resulting in unacceptably large receiver complexity for high-speed PAM data transmission on many channels. We propose a linear prefilter to force the overall impulse response of the channel/prefilter combination to approximate a desired truncated impulse response (DIR) of acceptably short duration. Given the duration of the DIR, the prefilter parameters and the DIR itself can be optimized adaptively to minimize the mean-square error between the output of the prefilter and the desired prefilter output, while constraining the energy in the DIR to be fixed. In this work we show that the minimum mean-square error can be expressed as the minimum eigenvalue of a certain channel-dependent matrix, and that the corresponding eigenvector represents the optimum DIR. An adaptive algorithm is developed and successfully tested. The simulations also show that the prefiltering scheme, used together with the VA for two different channel models, compares favorably in performance with another recently proposed prefiltering scheme. Limiting results for the case where the prefilter is considered to be of infinite length are obtained; it is shown that the optimum DIR of length two must be one of two possible impulse responses related to the duobinary impulse response. Finally we obtain limiting results for the case where the transmitting filter is optimized.