Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent approximate posterior distributions, and that acts as a stochastic encoder of the data. We develop stochastic back-propagation -- rules for back-propagation through stochastic variables -- and use this to develop an algorithm that allows for joint optimisation of the parameters of both the generative and recognition model. We demonstrate on several real-world data sets that the model generates realistic samples, provides accurate imputations of missing data and is a useful tool for high-dimensional data visualisation.

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Array processing involves manipulation of signals induced on various antenna elements. Its capabilities of steering nulls to reduce cochannel interferences and pointing independent beams toward various mobiles, as well as its ability to provide estimates of directions of radiating sources, make it attractive to a mobile communications system designer. Array processing is expected to play an important role in fulfilling the increased demands of various mobile communications services. Part I of this paper showed how an array could be utilized in different configurations to improve the performance of mobile communications systems, with references to various studies where feasibility of apt array system for mobile communications is considered. This paper provides a comprehensive and detailed treatment of different beam-forming schemes, adaptive algorithms to adjust the required weighting on antennas, direction-of-arrival estimation methods-including their performance comparison-and effects of errors on the performance of an array system, as well as schemes to alleviate them. This paper brings together almost all aspects of array signal processing.

/pdf/application-of-antenna-arrays-to-mobile-communications-ii-52fds1c81j.pdf

Application of antenna arrays to mobile communications. II. Beam-forming and direction-of-arrival considerations

Linear PA Design. Conventional High-Efficiency Amplifier Modes. Class AB PAs at GHz Frequencies. Practical Design of Class AB PAs. Overdrive and the Class F Mode. Switching Mode Amplifiers for RF Applications. Switching PA Modes at GHz Frequencies. Signals, Modulation Systems, and PA Nonlinearities. Efficiency Enhancement Techniques. Power Amplifier Bias Circuit Design. Power Amplifier Architecture. PA Linearization Techniques.

RF Power Amplifiers for Wireless Communications

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent an approximate posterior distribution and uses this for optimisation of a variational lower bound. We develop stochastic backpropagation - rules for gradient backpropagation through stochastic variables - and derive an algorithm that allows for joint optimisation of the parameters of both the generative and recognition models. We demonstrate on several real-world data sets that by using stochastic backpropagation and variational inference, we obtain models that are able to generate realistic samples of data, allow for accurate imputations of missing data, and provide a useful tool for high-dimensional data visualisation.

/pdf/stochastic-backpropagation-and-approximate-inference-in-deep-4qyrlossi0.pdf

Application of principles of statistical communication theory has led to a new communication system, called Rake, designed expressly to work against the combination of random multipath and additive noise disturbances. By coding the Mark-Space sequence of symbols to be transmitted into a wide-band signal, it becomes possible at the receiver to isolate those portions of the transmitted signal arriving with different delays, using correlation detection techniques. Before being recombined by addition, these separated signals are continuously and automatically processed so as to 1) apply to each an optimum weighting coefficient, derived from a measurement of the ionosphere response, and 2) introduce in each an appropriate delay such that they are all brought back into time coincidence. After a brief introduction, a functional description of the system is presented. There follows a review of the communication theory studies, which indicate that such systems have certain optimal properties. Details of design of an experimental prototype Rake system are followed by the results of limited field tests of this prototype. Conclusions and recommendations for future work are given.

A Communication Technique for Multipath Channels

If and only if the inputs to a set of nonlinear, zero-memory devices are variates drawn from a Gaussian random process, a useful general relationship may be found between certain input and output statistics of the set. This relationship equates partial derivatives of the (high-order) output correlation coefficient taken with respect to the input correlation coefficients, to the output correlation coefficient of a new set of nonlinear devices bearing a simple derivative relation to the original set. Application is made to the interesting special cases of conventional cross-correlation and autocorrelation functions, and Bussgang's theorem is easily proved. As examples, the output autocorrelation functions are simply obtained for a hard limiter, linear detector, clipper, and smooth limiter.

A useful theorem for nonlinear devices having Gaussian inputs

Solutions are obtained in open form for the optimum, probability-computing detector of either Gaussian signals, or known signals transmitted via scatter-paths, where the signals have been further perturbed by additive white Gaussian noise. The optimum receiver operates on the received waveforms with filter-functions and biasing constants determined by pairs of inhomogeneous and homogeneous integral equations, respectively. General solution in closed form has not been obtained, but it is possible to draw a few broad conclusions, among them that the filter-functions can be physically realizable. Approximate solution (for the optimum scatter-path receiver) at small signal-to-noise ratios yields a block diagram having interesting implications. For a single-scatter-path, the optimum receiver may be interpreted as the combination of a correlator with an optimum estimator of the Wiener type. Certain special cases in which complete solution is possible have been investigated in detail, and appropriate curves are presented. The role and performance of the probability-computing detector in an optimum decision-making receiver, for the types of channels considered, is deferred to a companion paper.

Optimum detection of random signals in noise, with application to scatter-multipath communication--I

Some non-central F-distributions expressed in closed form

The autocorrelation function and power spectrum of the phase-modulated portion of narrow-band gaussian noise is studied and specific computations are carried out for the case of a rectangular noise spectrum. Related results are given for the noise envelope. General asymptotic expressions for the spectra are obtained, together with experimental verification.

R. Price

Papers

A Communication Technique for Multipath Channels

A useful theorem for nonlinear devices having Gaussian inputs

Optimum detection of random signals in noise, with application to scatter-multipath communication--I

Some non-central F-distributions expressed in closed form

A note on the envelope and phase-modulated components of narrow-band Gaussian noise