Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Convex bodies: the brunn–minkowski theory

Consider a discrete source producing a sequence of message letters from a finite alphabet. A single-letter distortion measure is given by a non-negative matrix (d ij ). The entryd ij measures the ?cost? or ?distortion? if letteriis reproduced at the receiver as letterj. The average distortion of a communications system (source-coder-noisy channel-decoder) is taken to bed= ? i.j P ij d ij whereP ij is the probability ofibeing reproduced asj. It is shown that there is a functionR(d) that measures the ?equivalent rate? of the source for a given level of distortion. For coding purposes where a leveldof distortion can be tolerated, the source acts like one with information rateR(d). Methods are given for calculatingR(d), and various properties discussed. Finally, generalizations to ergodic sources, to continuous sources, and to distortion measures involving blocks of letters are developed. In this paper a study is made of the problem of coding a discrete source of information, given afidelity criterionor ameasure of the distortionof the final recovered message at the receiving point relative to the actual transmitted message. In a particular case there might be a certain tolerable level of distortion as determined by this measure. It is desired to so encode the information that the maximum possible signaling rate is obtained without exceeding the tolerable distortion level. This work is an expansion and detailed elaboration of ideas presented earlier [1], with particular reference to the discrete case. We shall show that for a wide class of distortion measures and discrete sources of information there exists a functionR(d) (depending on the particular distortion measure and source) which measures, in a sense, the equivalent rateRof the source (in bits per letter produced) whendis the allowed distortion level. Methods will be given for evaluatingR(d) explicitly in certain simple cases and for evaluatingR(d) by a limiting process in more complex cases. The basic results are roughly that it is impossible to signal at a rate faster thanC/R(d) (source letters per second) over a memoryless channel of capacityC(bits per second) with a distortion measure less than or equal tod. On the other hand, by sufficiently long block codes it is possible to approach as closely as desired the rateC/R(d) with distortion leveld. Finally, some particular examples, using error probability per letter of message and other simple distortion measures, are worked out in detail.

Coding Theorems for a Discrete Source With a Fidelity CriterionInstitute of Radio Engineers, International Convention Record, vol. 7, 1959.

We introduce a concentration property for probability measures on $\scriptstyle{R^n}$, which we call Property~($\scriptstyle\tau$); we show that this property has an interesting stability under products and contractions (Lemmas 1,~2,~3). Using property~($\scriptstyle\tau$), we give a short proof for a recent deviation inequality due to Talagrand. In a third section, we also recover known concentration results for Gaussian measures using our approach.}

Some deviation inequalities

The entropy power inequality, which plays a fundamental role in information theory and probability, may be seen as an analogue of the Brunn-Minkowski inequality. Motivated by this connection to Convex Geometry, we survey various recent developments on forward and reverse entropy power inequalities not just for the Shannon-Boltzmann entropy but also more generally for Renyi entropy. In the process, we discuss connections between the so-called functional (or integral) and probabilistic (or entropic) analogues of some classical inequalities in geometric functional analysis.

Forward and Reverse Entropy Power Inequalities in Convex Geometry

On the stability of Brunn–Minkowski type inequalities

An extension of the entropy power inequality to the form $N_{r}^\alpha (X+Y) \geq N_{r}^\alpha (X) + N_{r}^\alpha (Y)$ with arbitrary independent summands $X$ and $Y$ in $ {\mathbb {R}}^{n}$ is obtained for the Renyi entropy and powers $\alpha \geq ~(r+1)/2$ .

Variants of the Entropy Power Inequality

We derive a lower bound on the differential entropy of a log-concave random variable $X$ in terms of the $p$-th absolute moment of $X$. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. 
Specifically, we study the rate-distortion function for log-concave sources and distortion measure $| x - \hat x|^r$, and we establish that the difference between the rate distortion function and the Shannon lower bound is at most $\log(\sqrt{\pi e}) \approx 1.5$ bits, independently of $r$ and the target distortion $d$. For mean-square error distortion, the difference is at most $\log (\sqrt{\frac{\pi e}{2}}) \approx 1$ bits, regardless of $d$. 
We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most $\log (\sqrt{\frac{\pi e}{2}}) \approx 1$ bits. 
Our results generalize to the case of vector $X$ with possibly dependent coordinates, and to $\gamma$-concave random variables. Our proof technique leverages tools from convex geometry.

A lower bound on the differential entropy of log-concave random vectors with applications

In this paper we present new versions of the classical Brunn-Minkowski inequality for dierent classes of measures and sets. We show that the inequality (A + (1 )B) 1=n (A) 1=n + (1 ) (B) 1=n holds true for an unconditional product measure with decreasing density and a pair of unconditional convex bodies A;B R n . We also show that the above inequality is true for any unconditional logconcave measure and unconditional convex bodiesA;B R n . Finally, we prove that the inequality is true for a symmetric log-concave measure and a pair of symmetric convex sets A;B R 2 , which, in particular, settles two-dimensional case of the conjecture for Gaussian measure proposed in [13]. In addition, we deduce the 1=n-concavity of the parallel volume t7! (A +tB), Brunn’s type theorem and certain analogues of Minkowski first inequality.

/pdf/on-the-brunn-minkowski-inequality-for-general-measures-with-3cy9q2cs35.pdf

On the Brunn-Minkowski inequality for general measures with applications to new isoperimetric-type inequalities

We derive a lower bound on the differential entropy of a log-concave random variable X in terms of the p-th absolute moment of X. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. Specifically, we study the rate-distortion function for log-concave sources and distortion measure d
 (
 x
 ,
 x
 ^
 )
 =
 |
 x
 −
 x
 ^
 |
 r
 , with r
 ≥
 1
 , and we establish that the difference between the rate-distortion function and the Shannon lower bound is at most log
 (
 π
 e
 )
 ≈
 1
 .
 5
 bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most log
 (
 π
 e
 2
 )
 ≈
 1
 bit, regardless of d. We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most log
 (
 π
 e
 2
 )
 ≈
 1
 bit. Our results generalize to the case of a random vector X with possibly dependent coordinates. Our proof technique leverages tools from convex geometry.

https://www.mdpi.com/1099-4300/20/3/185/pdf

Arnaud Marsiglietti

Papers

On the stability of Brunn–Minkowski type inequalities

Variants of the Entropy Power Inequality

A lower bound on the differential entropy of log-concave random vectors with applications

On the Brunn-Minkowski inequality for general measures with applications to new isoperimetric-type inequalities

A Lower Bound on the Differential Entropy of Log-Concave Random Vectors with Applications.