Information Theory and Reliable Communication

1 Mean-field theory of phase transitions 2 Mean-field theory of spin glasses 3 Replica symmetry breaking 4 Gauge theory of spin glasses 5 Error-correcting codes 6 Image restoration 7 Associative memory 8 Learning in perceptron 9 Optimization problems A Eigenvalues of the Hessian B Parisi equation C Channel coding theorem D Distribution and free energy of K-Sat References Index

Statistical physics of spin glasses and information processing : an introduction

New lower bounds are presented for the minimum error probability that can be achieved through the use of block coding on noisy discrete memoryless channels. Like previous upper bounds, these lower bounds decrease exponentially with the block length N . The coefficient of N in the exponent is a convex function of the rate. From a certain rate of transmission up to channel capacity, the exponents of the upper and lower bounds coincide. Below this particular rate, the exponents of the upper and lower bounds differ, although they approach the same limit as the rate approaches zero. Examples are given and various incidental results and techniques relating to coding theory are developed. The paper is presented in two parts: the first, appearing here, summarizes the major results and treats the case of high transmission rates in detail; the second, to appear in the subsequent issue, treats the case of low transmission rates.

Lower Bounds to Error Probability for Coding on Discrete Memoryless Channels. I

In the low-energy high-energy-efficiency regime of classical optical communications — relevant to deep-space optical channels — there is a big gap between reliable communication rates achievable via conventional optical receivers and the ultimate (Holevo) capacity. Achieving the Holevo capacity requires not only optimal codes but also receivers that make collective measurements on long (modulated) codeword waveforms, and it is impossible to implement these collective measurements via symbol-by-symbol detection along with classical postprocessing [1], [2]. Here, we apply our recent results on the classical-quantum polar code [3] — the first near-explicit, linear, symmetric-Holevo-rate achieving code — to the lossy optical channel, and we show that it almost closes the entire gap to the Holevo capacity in the low photon number regime. In contrast, Arikan's original polar codes, applied to the DMC induced by the physical optical channel paired with any conceivable structured optical receiver (including optical homodyne, heterodyne, or direct-detection) fails to achieve the ultimate Holevo limit to channel capacity. However, our polar code construction (which uses the quantum fidelity as a channel parameter rather than the classical Bhattacharyya quantity to choose the “good channels” in the polar-code construction), paired with a quantum successive-cancellation receiver — which involves a sequence of collective non-destructive binary projective measurements on the joint quantum state of the received codeword waveform — can attain the Holevo limit, and can hence in principle achieve higher rates than Arikan's polar code and decoder directly applied to the optical channel. However, even a theoretical recipe for construction of an optical realization of the quantum successive-cancellation receiver remains an open question.

Polar coding to achieve the Holevo capacity of a pure-loss optical channel

The stochastic block model (SBM) is a popular framework for studying community detection in networks. This model is limited by the assumption that all nodes in the same community are statistically equivalent and have equal expected degrees. The degree-corrected stochastic block model (DCSBM) is a natural extension of SBM that allows for degree heterogeneity within communities. This paper proposes a convexified modularity maximization approach for estimating the hidden communities under DCSBM. Our approach is based on a convex programming relaxation of the classical (generalized) modularity maximization formulation, followed by a novel doubly-weighted $ \ell_1 $-norm $ k $-median procedure. We establish non-asymptotic theoretical guarantees for both approximate clustering and perfect clustering. Our approximate clustering results are insensitive to the minimum degree, and hold even in sparse regime with bounded average degrees. In the special case of SBM, these theoretical results match the best-known performance guarantees of computationally feasible algorithms. Numerically, we provide an efficient implementation of our algorithm, which is applied to both synthetic and real-world networks. Experiment results show that our method enjoys competitive performance compared to the state of the art in the literature.

Convexified Modularity Maximization for Degree-corrected Stochastic Block Models

In this paper, we propose an open-loop unequal-error-protection querying policy based on superposition coding for the noisy 20 questions problem. In this problem, a player wishes to successively refine an estimate of the value of a continuous random variable by posing binary queries and receiving noisy responses. When the queries are designed non-adaptively as a single block and the noisy responses are modeled as the output of a binary symmetric channel, the 20 questions problem can be mapped to an equivalent problem of channel coding with unequal error protection (UEP). A new non-adaptive querying strategy based on UEP superposition coding is introduced, whose estimation error decreases with an exponential rate of convergence that is significantly better than that of the UEP repetition coding introduced by Variani et al. (2015). With the proposed querying strategy, the rate of exponential decrease in the number of queries matches the rate of a closed-loop adaptive scheme, where queries are sequentially designed with the benefit of feedback. Furthermore, the achievable error exponent is significantly better than that of random block codes employing equal error protection.

Unequal Error Protection Querying Policies for the Noisy 20 Questions Problem

We show a direct relationship between the variance and the differential entropy for subclasses of symmetric and asymmetric unimodal distributions by providing an upper bound on variance in terms of entropy power. Combining this bound with the well-known entropy power lower bound on variance, we prove that the variance of the appropriate subclasses of unimodal distributions can be bounded below and above by the scaled entropy power. As the differential entropy decreases, the variance is sandwiched between two exponentially decreasing functions in the differential entropy. This establishes that for the subclasses of unimodal distributions, the differential entropy can be used as a surrogate for concentration of the distribution.

Bounds on Variance for Unimodal Distributions

Optical channel with direct detections, also known as Poisson channel, is well studied. With direction detections, only the intensity of the optical signal is measured and used as the carrier of information. In coherent detection, the phase of optical signal is also utilized. Constrained by optical devices, it is hard to directly measure this phase. However, it is possible to mix the input signal with a local reference signal to create output that is phase dependent. Generating such local signals can be based on the instantaneous receiver knowledge, and updated from time to time. We study the channel capacity of such channel, and make connection to the recent development in feedback channels. We are particularly interested in the low SNR regime, and present a capacity result based on a new scaling law.

On capacity of optical channels with coherent detection

We show a direct relationship between the variance and the differential entropy for subclasses of symmetric and asymmetric unimodal distributions by providing an upper bound on variance in terms of entropy power. Combining this bound with the well-known entropy power lower bound on variance, we prove that the variance of the appropriate subclasses of unimodal distributions can be bounded below and above by the scaled entropy power. As differential entropy decreases, the variance is sandwiched between two exponentially decreasing functions in the differential entropy. This establishes that for the subclasses of unimodal distributions, the differential entropy can be used as a surrogate for concentration of the distribution.

We consider query-based data acquisition and the corresponding information recovery problem, where the goal is to recover $k$ binary variables (information bits) from parity measurements of those variables. The queries and the corresponding parity measurements are designed using the encoding rule of Fountain codes. By using Fountain codes, we can design potentially limitless number of queries, and corresponding parity measurements, and guarantee that the original $k$ information bits can be recovered with high probability from any sufficiently large set of measurements of size n. In the query design, the average number of information bits that is associated with one parity measurement is called query difficulty $(\bar{d})$ and the minimum number of measurements required to recover the $k$ information bits for a fixed $\bar{d}$ is called sample complexity (n). We analyze the fundamental trade-offs between the query difficulty and the sample complexity, and show that the sample complexity of $n=c\max\{k,(k \log k)/\bar{d}\}$ for some constant $c$ > 0 is necessary and sufficient to recover $k$ information bits with high probability as $k\rightarrow\infty$ .1 A full version of this paper is accessible at arXiv:I712.00157 [I]

Hye Won Chung

Papers

Unequal Error Protection Querying Policies for the Noisy 20 Questions Problem

Bounds on Variance for Unimodal Distributions

On capacity of optical channels with coherent detection

Bounds on Variance for Unimodal Distributions

Fundamental Limits on Data Acquisition: Trade-offs Between Sample Complexity and Query Difficulty