This comprehensive treatment of network information theory and its applications provides the first unified coverage of both classical and recent results. With an approach that balances the introduction of new models and new coding techniques, readers are guided through Shannon's point-to-point information theory, single-hop networks, multihop networks, and extensions to distributed computing, secrecy, wireless communication, and networking. Elementary mathematical tools and techniques are used throughout, requiring only basic knowledge of probability, whilst unified proofs of coding theorems are based on a few simple lemmas, making the text accessible to newcomers. Key topics covered include successive cancellation and superposition coding, MIMO wireless communication, network coding, and cooperative relaying. Also covered are feedback and interactive communication, capacity approximations and scaling laws, and asynchronous and random access channels. This book is ideal for use in the classroom, for self-study, and as a reference for researchers and engineers in industry and academia.

Network Information Theory

Information Theory and Reliable Communication

Sphere Packings, Lattices and Groups

Systems and methods for implementing array cameras configured to perform super- resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. Lens stack arrays in accordance with many embodiments of the invention include lens elements formed on substrates separated by spacers, where the lens elements, substrates and spacers are configured to form a plurality of optical channels, at least one aperture located within each optical channel, at least one spectral filter located within each optical channel, where each spectral filter is configured to pass a specific spectral band of light, and light blocking materials located within the lens stack array to optically isolate the optical channels.

Capturing and processing of images using monolithic camera array with heterogeneous imagers

Product design is a highly involved, often ill-defined, complex and iterative process, and the needs and specifications of the required artifact get more refined only as the design process moves toward its goal. An effective computer support tool that helps the designer make better-informed decisions requires efficient knowledge representation schemes. In today's world, there is a virtual explosion in the amount of raw data available to the designer, and knowledge representation is critical in order to sift through this data and make sense of it. In addition, the need to stay competitive has shrunk product development time through the use of simultaneous and collaborative design processes, which depend on effective transfer of knowledge between teams. Finally, the awareness that decisions made early in the design process have a higher impact in terms of energy, cost, and sustainability, has resulted in the need to project knowledge typically required in the later stages of design to the earlier stages. Research in design rationale systems, product families, systems engineering, and ontology engineering has sought to capture knowledge from earlier product design decisions, from the breakdown of product functions and associated physical features, and from customer requirements and feedback reports. VR (Virtual reality) systems and multidisciplinary modeling have enabled the simulation of scenarios in the manufacture, assembly, and use of the product. This has helped capture vital knowledge from these stages of the product life and use it in design validation and testing. While there have been considerable and significant developments in knowledge capture and representation in product design, it is useful to sometimes review our position in the area, study the evolution of research in product design, and from past and current trends, try and foresee future developments. The goal of this paper is thus to review both our understanding of the field and the support tools that exist for the purpose, and identify the trends and possible directions research can evolve in the future.

/pdf/the-evolution-challenges-and-future-of-knowledge-2l8tzvf3b5.pdf

The evolution, challenges, and future of knowledge representation in product design systems

We analyze the performance of the Borda counting algorithm in a non-parametric model. The algorithm needs to utilize probabilistic rankings of the items within <inline-formula><tex-math notation="LaTeX">$m$</tex-math></inline-formula>-sized subsets to accurately determine which items are the overall top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> items in a total of <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> items. The Borda counting algorithm simply counts the cumulative scores for each item from these partial ranking observations. This generalizes a previous work of a similar nature by Shah <italic>et al.</italic> using probabilistic pairwise comparison data. The performance of the Borda counting algorithm critically depends on the associated score separation <inline-formula><tex-math notation="LaTeX">$\Delta _{k}$</tex-math></inline-formula> between the <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula>-th item and the <inline-formula><tex-math notation="LaTeX">$(k+1)$</tex-math></inline-formula>-th item. Specifically, we show that if <inline-formula><tex-math notation="LaTeX">$\Delta _{k}$</tex-math></inline-formula> is greater than certain value, then the top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> items selected by the algorithm is asymptotically accurate almost surely; if <inline-formula><tex-math notation="LaTeX">$\Delta _{k}$</tex-math></inline-formula> is below certain value, then the result will be inaccurate with a constant probability. In the special case of <inline-formula><tex-math notation="LaTeX">$m=2$</tex-math></inline-formula>, i.e., pairwise comparison, the resultant bound is tighter than that given by Shah <italic>et al.</italic>, leading to a reduced gap between the error probability upper and lower bounds. These results are further extended to the approximate top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> selection setting. Numerical experiments demonstrate the effectiveness and accuracy of the Borda counting algorithm, compared with the spectral MLE-based algorithm, particularly when the data does not necessarily follow an assumed parametric model.

On Top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> Selection From <inline-formula><tex-math notation="LaTeX">$m$</tex-math></inline-formula>-Wise Partial Rankings via Borda Counting

The problem of multilevel diversity coding with secure regeneration (MDC-SR) is considered, which includes the problems of multilevel diversity coding with regeneration (MDC-R) and secure regenerating code (SRC) as special cases. Two outer bounds are established, showing that separate coding of different messages using the respective SRCs can achieve the minimum-bandwidth-regeneration (MBR) point of the achievable normalized storage-capacity repair-bandwidth tradeoff regions for the general MDC-SR problem. The core of the new converse results is an exchange lemma, which can be established using Han's subset inequality.

/pdf/multilevel-diversity-coding-with-regeneration-separate-qbyxjq174i.pdf

Multilevel Diversity Coding with Regeneration: Separate Coding Achieves the MBR Point

We provide a new information-theoretic generalization error bound that is exactly tight (i.e., matching even the constant) for the canonical quadratic Gaussian mean estimation problem. Despite considerable existing efforts in deriving information-theoretic generalization error bounds, applying them to this simple setting where sample average is used as the estimate of the mean value of Gaussian data has not yielded satisfying results. In fact, most existing bounds are order-wise loose in this setting, which has raised concerns about the fundamental capability of information-theoretic bounds in reasoning the generalization behavior for machine learning. The proposed new bound adopts the individual-sample-based approach proposed by Bu et al., but also has several key new ingredients. Firstly, instead of applying the change of measure inequality on the loss function, we apply it to the generalization error function itself; secondly, the bound is derived in a conditional manner; lastly, a reference distribution, which bears a certain similarity to the prior distribution in the Bayesian setting, is introduced. The combination of these components produces a general KL-divergence-based generalization error bound. We further show that although the conditional bounding and the reference distribution can make the bound exactly tight, removing them does not significantly degrade the bound, which leads to a mutual-information-based bound that is also asymptotically tight in this setting.

Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problem

We consider information leakage to the user in private information retrieval (PIR) systems. Information leakage can be measured in terms of individual message leakage or total leakage. Individual message leakage, or simply individual leakage, is defined as the amount of information that the user can obtain on any individual message that is not being requested, and the total leakage is defined as the amount of information that the user can obtain about all the other messages except the one being requested. In this work, we characterize the tradeoff between the minimum download cost and the individual leakage, and that for the total leakage, respectively. Coding schemes are proposed to achieve these optimal tradeoffs, which are also shown to be optimal in terms of the message size. We further characterize the optimal tradeoff between the minimum amount of common randomness and the total leakage. Moreover, we show that under individual leakage, common randomness is in fact unnecessary when there are more than two messages.

On the Information Leakage in Private Information Retrieval Systems

The problem of multilevel diversity coding with secure regeneration is revisited. Under the assumption that the eavesdropper can access the repair data for all compromised storage nodes, Shao el al. provided a precise characterization of the minimum-bandwidth-regeneration (MBR) point of the achievable normalized storage-capacity repair-bandwidth tradeoff region. In this paper, it is shown that the MBR point of the achievable normalized storage-capacity repair-bandwidth tradeoff region remains the same even if we assume that the eavesdropper can access the repair data for some compromised storage nodes (type II compromised nodes) but only the data contents of the remaining compromised nodes (type I compromised nodes), as long as the number of type I compromised nodes is no greater than that of type II compromised nodes.

Chao Tian

Papers

On Top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> Selection From <inline-formula><tex-math notation="LaTeX">$m$</tex-math></inline-formula>-Wise Partial Rankings via Borda Counting

Multilevel Diversity Coding with Regeneration: Separate Coding Achieves the MBR Point

Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problem

On the Information Leakage in Private Information Retrieval Systems

New Results on Multilevel Diversity Coding with Secure Regeneration