Showing papers presented at "Information Theory Workshop in 2015"

Deep learning and the information bottleneck principle

Abstract: Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.

Optimal linear and cyclic locally repairable codes over small fields

Abstract: We consider locally repairable codes over small fields and propose constructions of optimal cyclic and linear codes in terms of the dimension for a given distance and length.

Upper bound on the capacity of a cascade of nonlinear and noisy channels

Abstract: An upper bound on the capacity of a cascade of nonlinear and noisy channels is presented. The cascade mimics the split-step Fourier method for computing waveform propagation governed by the stochastic generalized nonlinear Schrodinger equation. It is shown that the spectral efficiency of the cascade is at most log(1+SNR), where SNR is the receiver signal-to-noise ratio. The results may be applied to optical fiber channels. However, the definition of bandwidth is subtle and leaves open interpretations of the bound. Some of these interpretations are discussed.

Cyclic linear binary locally repairable codes

Abstract: Locally repairable codes (LRCs) are a class of codes designed for the local correction of erasures. They have received considerable attention in recent years due to their applications in distributed storage. Most existing results on LRCs do not explicitly take into consideration the field size q, i.e., the size of the code alphabet. In particular, for the binary case, only a few specific results are known by Goparaju and Calderbank. Recently, however, an upper bound on the dimension k of LRCs was presented by Cadambe and Mazumdar. The bound takes into account the length n, minimum distance d, locality r, and field size q, and it is applicable to both non-linear and linear codes. In this work, we first develop an improved version of the bound mentioned above for linear codes. We then focus on cyclic linear binary codes. By leveraging the cyclic structure, we notice that the locality of such a code is determined by the minimum distance of its dual code. Using this result, we investigate the locality of a variety of well known cyclic linear binary codes, e.g., Hamming codes and Simplex codes, and also prove their optimality with our improved bound for linear codes. We also discuss the locality of codes which are obtained by applying the operations of Extend, Shorten, Expurgate, Augment, and Lengthen to cyclic linear binary codes. Several families of such modified codes are considered and their optimality is addressed. Finally, we investigate the locality of Reed-Muller codes. Even though they are not cyclic, it is shown that some of the locality results for cyclic codes still apply.

Construction of polar codes for channels with memory

Abstract: Polar codes for channels with memory are considered in this paper. Sasoglu proved that Arikan's polarization applies to finite-state memory channels but practical decoding algorithms of polar codes for such channels have been unknown. This paper proposes a generalization of successive cancellation algorithm for channels with memory where the complexity is polynomial in the number of states. In addition, two polar coding scheme are proposed to generate codewords following non-i.i.d. process required to achieve the capacity. Whereas one is a simple application of the polar coding scheme for asymmetric memoryless channels, the other one combines a polar code with a fixed-to-variable length homophonic coding scheme, which can realize the input distribution exactly equal to the target process.

Caching-aided coded multicasting with multiple random requests

Abstract: The capacity of caching networks has received considerable attention in the past few years. A particularly studied setting is the shared link caching network, in which a single source with access to a file library communicates with multiple users, each having the capability to store segments (packets) of the library files, over a shared multicast link. Each user requests one file from the library according to a common demand distribution and the server sends a coded multicast message to satisfy all users at once. The problem consists of finding the smallest possible average codeword length to satisfy such requests. In this paper, we consider the generalization to the case where each user places L ≥ 1 independent requests according to the same common demand distribution. We propose an achievable scheme based on random vector (packetized) caching placement and multiple groupcast index coding, shown to be order-optimal in the asymptotic regime in which the number of packets per file B goes to infinity. We then show that the scalar (B = 1) version of the proposed scheme can still preserve order-optimality when the number of per-user requests L is large enough. Our results provide the first order-optimal characterization of the shared link caching network with multiple random requests, revealing the key effects of L on the performance of caching-aided coded multicast schemes.

Achieving secrecy capacity of the wiretap channel and broadcast channel with a confidential component

Abstract: We show that capacity of the general (not necessarily degraded or symmetric) wiretap channel under a “strong secrecy constraint” can be achieved using an explicit scheme based on polar codes. We also extend our construction to the case of broadcast channels with confidential messages defined by Csiszar and Korner, achieving the entire capacity region of this communication model. This submission is an extended abstract of the paper by the same authors (see arXiv:1410.3422).

Polar coding for the general wiretap channel

Abstract: Information-theoretic work for wiretap channels is mostly based on random coding schemes. Designing practical coding schemes to achieve information-theoretic security is an important problem. By applying two recently developed techniques for polar codes, namely, universal polar coding and polar coding for asymmetric channels, we propose a polar coding scheme to achieve the secrecy capacity of the general wiretap channel.

Privacy on hypothesis testing in smart grids

Abstract: In this paper, we study the problem of privacy information leakage in a smart grid. The privacy risk is assumed to be caused by an unauthorized binary hypothesis testing of the consumer's behaviour based on the smart meter readings of energy supplies from the energy provider. Another energy supplies are produced by an alternative energy source. A controller equipped with an energy storage device manages the energy inflows to satisfy the energy demand of the consumer. We study the optimal energy control strategy which minimizes the asymptotic exponential decay rate of the minimum Type II error probability in the unauthorized hypothesis testing to suppress the privacy risk. Our study shows that the cardinality of the energy supplies from the energy provider for the optimal control strategy is no more than two. This result implies a simple objective of the optimal energy control strategy. When additional side information is available for the adversary, the optimal control strategy and privacy risk are compared with the case of leaking smart meter readings to the adversary only.

Information-theory-friendly models for fiber-optic channels: A primer

Abstract: There exists a rich flora of channel models for optical fiber channels, which differ not only in the types of transmission scenario they describe but also in the type of analysis they support. In this tutorial paper, we review several channel models used in optical communications, and discuss their suitability for information-theoretic analyses. Key issues are how nonlinearity, channel memory, and multiuser interference are modeled.

Lattice code design criterion for MIMO wiretap channels

Abstract: We consider the MIMO wiretap channel, where a legitimate transmitter (Alice) wishes to communicate a confidential message to the legitimate receiver (Bob) over a MIMO channel, while its messages are being eavesdropped by Eve through another MIMO channel. Supposing that Alice uses the nested lattice code proposed in [8], we characterize the expected value of amount of information leakage to Eve, and from minimizing this expected value, we derive a code design criterion for MIMO lattice wiretap codes. The special cases of block and fast fading channels are also studied. We also illustrate our lattice code design criterion using the Alamouti Code as the simplest MIMO code available in the literature.

Exact recovery threshold in the binary censored block model

Abstract: Given a background graph with n vertices, the binary censored block model assumes that vertices are partitioned into two clusters, and every edge is labeled independently at random with labels drawn from Bern(1 − e) if two endpoints are in the same cluster, or from Bern(e) otherwise, where e ∈ [0; 1/2] is a fixed constant. For Erdős-Renyi graphs with edge probability p = a log n/n and fixed a, we show that the semidefinite programming relaxation of the maximum likelihood estimator achieves the optimal threshold equation for exactly recovering the partition from the labeled graph with probability tending to one as n → ∞. For random regular graphs with degree scaling as a log n, we show that the semidefinite programming relaxation also achieves the optimal recovery threshold aD(Bern(1/2)∥Bern(e)) > 1, where D denotes the Kullback-Leibler divergence.

Index coding and network coding via rank minimization

Abstract: Index codes reduce the number of bits broadcast by a wireless transmitter to a number of receivers with different demands and with side information. It is known that the problem of finding optimal linear index codes is NP-hard. We investigate the performance of different heuristics based on rank minimization and matrix completion methods for constructing linear index codes over the reals. As a summary of our results, the alternating projections method gives the best results in terms of minimizing the number of broadcast bits and convergence rate and leads to up to 13% savings in communication cost compared to graph coloring algorithms studied in the literature. Moreover, we describe how the proposed methods can be used to construct linear network codes for non-multicast networks.

Strategic compression and transmission of information

Abstract: This paper considers the problem of communication in the context of strategic information transfer (SIT) concept of Crawford and Sobel in economics. SIT is different from the conventional communication paradigms since it involves different objectives for the encoder and the decoder, which are aware of this mismatch and act accordingly. This leads to a game whose equilibrium solutions are studied here. We model the problem as a Stackelberg game-as opposed to the Nash model used in prior work in economics- where the encoder is the leader and its distortion measure depends on a private information sequence which is non-causally available, only to the encoder; and the decoder is the follower. We consider three problem settings focusing on the quadratic distortion measures and jointly Gaussian source and private information: compression, communication (joint source-channel coding over a scalar Gaussian channel), and the simple equilibrium conditions without any compression or communication. We characterize the fundamental limits -asymptotic in blocklength- of the equilibrium strategies and associated costs for these problems. For the quadratic-Gaussian case, we compute the equilibrium conditions and strategic rate-distortion function explicitly, and show optimality of uncoded communication over an additive white Gaussian channel, paralleling the well-known optimality of uncoded communication in the conventional, “nonstrategic” communication setting.

Improved layered regenerating codes characterizing the exact-repair storage-repair bandwidth tradeoff for certain parameter sets

Abstract: The characterization of the storage-repair bandwidth tradeoff of (n, k, d)-regenerating codes under the exact-repair setting remains an open problem. The problem has been solved only for the special case of (n, k, d) = (4, 3, 3). In the present paper, we characterize the tradeoff for the larger family of parameters (n, k = 3, d = n − 1). This is accomplished by constructing an (n, k < d, d)-regenerating code, referred to as the improved layered code. In the case when (n, k = 3, d = n − 1), the code operates on a point that coincides with an interior point of a recently derived outer bound on the tradeoff. The code also achieves an interior point on the outer bound for the parameter set (n, k = 4, d = n − 1).

The wiretap channel with generalized feedback: Secure communication and key generation

Abstract: It is well-known that feedback does not increase the capacity of point-to-point memoryless channels, however, the advantage it poses to secure communications is not fully understood yet. In this work, an achievable scheme for the wiretap channel with generalized feedback is presented. This scheme employs the feedback signal to generate a secret key shared between the legitimate users, which is then used to encrypt the message to be sent. It is shown that the resulting achievable rate recovers previous results, thus it can be seen as a generalization and unification of several results in the field.

Coordinating partially-informed agents over state-dependent networks

Abstract: We consider a multi-agent scenario with K ≥ 2 agents that have partial information about some random nature state, and that take actions in a repeated manner. Each agent also has imperfect observations of the other agents' past actions and the nature state realization. Our goal is to characterize the set of asymptotically implementable distributions on the agents' actions and the nature state. We solve this problem for general K when all agents have only causal nature state information (NSI) and for K = 2 when: one agent has causal NSI and the other agent has non-causal NSI; or in some special cases when both agents have non-causal NSI.

Secret-key cryptography from ideal primitives: A systematic overview

Abstract: Secret-key constructions are often proved secure in a model where one or more underlying components are replaced by an idealized oracle accessible to the attacker. This model gives rise to information-theoretic security analyses, and several advances have been made in this area over the last few years. This paper provides a systematic overview of what is achievable in this model, and how existing works fit into this view.

Codes for DNA storage channels

Abstract: We consider the problem of assembling a sequence based on a collection of its substrings observed through a noisy channel. This problem of reconstructing sequences from traces was first investigated in the noiseless setting under the name of “Markov type” analysis. Here, we explain the connection between the problem and the problem of DNA synthesis and sequencing, and introduce the notion of a DNA storage channel. We analyze the number of sequence equivalence classes under the channel mapping and propose new asymmetric coding techniques to combat the effects of synthesis noise. In our analysis, we make use of Ehrhart theory for rational polytopes.

Trellis based lower bounds on capacities of channels with synchronization errors

Abstract: In this paper, we model synchronization error channels as hidden Markov models. Through this model we introduce a rhomboidal trellis for calculating information rates for synchronization error channels. In particular, the introduced trellis is novel due to being synchronized with the channel input sequence, i.e., one trellis section corresponds to one channel input symbol, but (possibly) multiple channel output symbols. By further conditioning on the output sequence length, we construct a finite-length/finite-width trellis, and proceed to provide Monte Carlo methods to evaluate the information rates for Markov channel inputs of any finite order. We also demonstrate how to optimize the Markov process using a generalized Blahut-Arimoto technique to improve on the known lower bounds for the deletion channel.

Talking reliably, secretly, and efficiently: A “complete” characterization

Abstract: We consider reliable and secure communication of information over a multipath network. A transmitter Alice sends messages to the receiver Bob in the presence of a hidden adversary Calvin. The adversary Calvin can both eavesdrop and jam on (possibly non-identical) subsets of transmission links. The goal is to communicate reliably (intended receiver can understand the messages) and secretly (adversary cannot understand the messages). Two kinds of jamming, additive and overwrite, are considered. Additive jamming corresponds to wireless network model while overwrite jamming corresponds to wired network model and storage systems. The multipath network consists of C parallel links. Calvin can both jam and eavesdrop any z io number of links, can eavesdrop (but not jam) any z i/o number of links, and can jam (but not eavesdrop) any z o/i number of links. We present the first “complete” information-theoretic characterization of maximum achievable rate as a function of the number of links that can be jammed and/or eavesdropped for equal and unequal link capacity multipath networks under additive and overwrite jamming in the large alphabet regime. Our achievability and converse proofs require non-trivial combination of information theoretic and coding theoretic ideas and our achievability schemes are computationally efficient. The PHaSE-Saving techniques1 are used for achievability while a “stochastic” singleton bound is obtained for converse.

Capacity of Binary State Symmetric Channel with and without feedback and transmission cost

Abstract: We consider a unit memory channel, called Binary State Symmetric Channel (BSSC), in which the channel state is the modulo2 addition of the current channel input and the previous channel output. We derive closed form expressions for the capacity and corresponding channel input distribution for the BSSC with and without feedback and transmission cost. We also show that the capacity of the BSSC, with or without feedback, is achieved by a first order symmetric Markov process.

Asymmetric Lee distance codes: New bounds and constructions

Abstract: We continue our study of a new family of asymmetric Lee codes that arise in the design and implementation of emerging DNA-based storage systems and systems which use parallel string transmission protocols The codewords are defined over a quaternary alphabet, although the results carry over to other alphabet sizes, and have symbol distances dictated by their underlying binary representation Our contributions include deriving new bounds for the size of the largest code in this metric based on Delsarte-like linear programming methods and describing new constructions for non-linear asymmetric Lee codes

Upper bounds on the relative entropy and Rényi divergence as a function of total variation distance for finite alphabets

Abstract: A new upper bound on the relative entropy is derived as a function of the total variation distance for probability measures defined on a common finite alphabet. The bound improves a previously reported bound by Csiszar and Talata. It is further extended to an upper bound on the Renyi divergence of an arbitrary non-negative order (including ∞) as a function of the total variation distance.

A mixed-ADC receiver architecture for massive MIMO systems

Abstract: Motivated by the demand for energy-efficient communication solutions in the next generation cellular network, a mixed-ADC receiver architecture for massive multiple input multiple output (MIMO) systems is proposed, which differs from previous works in that herein one-bit analog-to-digital converters (ADCs) partially replace the conventionally assumed high-resolution ADCs. The information-theoretic tool of generalized mutual information (GMI) is exploited to analyze the achievable data rates of the proposed system architecture and an array of analytical results of engineering interest are obtained. For deterministic single input multiple output (SIMO) channels, a closed-form expression of the GMI is derived, based on which the linear combiner is optimized. Then, the asymptotic behaviors of the GMI in both low and high SNR regimes are explored, and the analytical results suggest a plausible ADC assignment scheme. Finally, the analytical framework is applied to the multi-user access scenario, and the corresponding numerical results demonstrate that the mixed system architecture with a relatively small number of high-resolution ADCs is able to achieve a large fraction of the channel capacity without output quantization.

Strong coordination over multi-hop line networks

Abstract: We analyze the problem of strong coordination over a multi-hop line network where the node initiating the coordination is a terminal network node. We provide a characterization of the capacity region when the initiating node possesses unlimited local randomness and intermediate nodes operate under a functional regime. In this regime, next-hop messages are created only using common randomness and previous-hop incoming messages, i.e., local randomness at intermediate nodes is only used for generating actions.

Repair scheduling in wireless distributed storage with D2D communication

Abstract: We consider distributed storage (DS) for a wireless network where mobile devices arrive and depart according to a Poisson random process. Content is stored in a number of mobile devices, using an erasure correcting code. When requesting a piece of content, a user retrieves the content from the mobile devices using device-to-device communication or, if not possible, from the base station (BS), at the expense of a higher communication cost. We consider the repair problem when a device that stores data leaves the network. In particular, we introduce a repair scheduling where repair is performed (from storage devices or the BS) periodically. We derive analytical expressions for the overall communication cost of repair and download as a function of the repair interval. We illustrate the analysis by giving results for maximum distance separable codes and regenerating codes. Our results indicate that DS can reduce the overall communication cost with respect to the case where content is only downloaded from the BS, provided that repairs are performed frequently enough. The required repair frequency depends on the code used for storage and the network parameters. In particular, minimum bandwidth regenerating codes require frequent repairs, while maximum distance separable codes give better performance if repair is performed less frequently. We also show that instantaneous repair is not always optimal.

Tight bounds for symmetric divergence measures and a new inequality relating f-divergences

Abstract: Tight bounds for several symmetric divergence measures are introduced, given in terms of the total variation distance. Each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszar. A new inequality relating f-divergences is derived, and its use is exemplified. The last section of this conference paper is not included in the recent journal paper [16], as well as some new remarks that are linked to new references.

Generalized interlinked cycle cover for index coding

Abstract: A source coding problem over a noiseless broadcast channel where the source is preinformed about the contents of the cache of all receivers, is an index coding problem. Furthermore, if each message is requested by one receiver, then we call this an index coding problem with a unicast message setting. This problem can be represented by a directed graph. In this paper, we first define a structure (we call generalized interlinked cycle (GIC)) in directed graphs. A GIC consists of cycles which are interlinked in some manner (i.e., not disjoint), and it turns out that the GIC is a generalization of cliques and cycles. We then propose a simple scalar linear encoding scheme with linear time encoding complexity. This scheme exploits GICs in the digraph. We prove that our scheme is optimal for a class of digraphs with message packets of any length. Moreover, we show that our scheme can outperform existing techniques, e.g., partial clique cover, local chromatic number, composite-coding, and interlinked cycle cover.

Non-binary GLD codes and their lattices

Abstract: The recently discovered family of generalized low-density (GLD) lattices brings new mathematical challenges to coding theorists and practitioners. Given the excellent performance of integer GLD lattices in high dimensions and motivated by the simple lattice structure used for fast iterative decoding, this paper is a first attempt to analyze GLD lattices for asymptotically large dimensions. Firstly, we describe non-binary GLD codes and show their asymptotic goodness in terms of minimum Hamming distance. Secondly, we consider a GLD lattice ensemble built via Construction A from non-binary GLD codes, and analyze their goodness with respect to Poltyrev limit on the Gaussian channel. Finally, at large dimensions and using a large code alphabet, we prove that infinite GLD lattice constellations attain Poltyrev capacity limit under maximum likelihood decoding.