This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose SNR is chosen uniformly distributed between 0 and SNR.

/pdf/mutual-information-and-minimum-mean-square-error-in-gaussian-335yeawbxb.pdf

Mutual information and minimum mean-square error in Gaussian channels

This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose signal-to-noise ratio is chosen uniformly distributed between 0 and SNR.

Mutual Information and Minimum Mean-square Error in Gaussian Channels

The issue of fundamental limitations in filtering and control lies at the very heart of any feedback system design, since it reveals what is and is not achievable on the basis of that system's structural and dynamic characteristics Alongside new succinct treatments of Bode's original results from the 1940s, this book presents a comprehensive analysis of modern results, featuring contemporary developments in multivariable systems, sampled-data, periodic and nonlinear problems The text gives particular prominence to sensitivity functions which measure the fundamental qualities of the system, including performance and robustness With extensive appendices covering the necessary background on complex variable theory, this book is an ideal self-contained resource for researchers and practitioners in this field

Fundamental Limitations in Filtering and Control

A survey of data smoothing for linear and nonlinear dynamic systems

Discrete-time fixed-lag smoothing algorithms

The fixed-lag smoother as a stable finite-dimensional linear system

Comparison is made between the mean-square errors Pf and Ps associated with the linear least-squares filtered and smoothed estimates of a stationary process of spectral density S(ω) in white noise of spectral density No. A universal curve is obtained which relates the minimum possible value of Ps/Pf to ωmax{S(ω) No}. The curve sets a bound on the maximum improvement over filtering which smoothing will offer, in terms of the maximum signal/noise ratio.

Smoothing as an improvement on filtering: a universal bound

Attempts to realize optimal fixed-lag smoothers for continuous-time processes have encountered two major difficulties: the presence of delay lines in the smoother, and the instability of most smoothers. This paper considers suboptimal smoothers which still outperform filters and have performance very close to the optimum. Design procedures are exhibited for smoothers that are both finite-dimensional and asymptotically stable, with the procedures depending on the delay line being approximated by a finite-dimensional dynamical system.

Stable fixed-lag smoothing of continuous time processes

Formulas are given expressing the smoothed estimate of the state of a noisy linear system in terms of filtered estimates of the state, for both continuous and discrete time.

https://ieeexplore.ieee.org/iel5/9/24123/01099901.pdf

New linear smoothing formulas

Three basic methods for constructing suboptimal continuous-time linear fixed-lag smoothers are described. In contrast to the optimal smoother, all the suboptimal smoothers are stable. The degree of complexity allowed in the suboptimal smoothers determines how closely they perform to the limit obtainable from the theoretical optimal linear fixed-lag smoother.

S. Chirarattananon

Papers

The fixed-lag smoother as a stable finite-dimensional linear system

Smoothing as an improvement on filtering: a universal bound

Stable fixed-lag smoothing of continuous time processes

New linear smoothing formulas

Outline designs for stable continuous-time fixed-lag smoothers