1 Introduction.- 2 Experiments, Deficiencies, Distances v.- 2.1 Comparing Risk Functions.- 2.2 Deficiency and Distance between Experiments.- 2.3 Likelihood Ratios and Blackwell's Representation.- 2.4 Further Remarks on the Convergence of Distri butions of Likelihood Ratios.- 2.5 Historical Remarks.- 3 Contiguity - Hellinger Transforms.- 3.1 Contiguity.- 3.2 Hellinger Distances, Hellinger Transforms.- 3.3 Historical Remarks.- 4 Gaussian Shift and Poisson Experiments.- 4.1 Introduction.- 4.2 Gaussian Experiments.- 4.3 Poisson Experiments.- 4.4 Historical Remarks.- 5 Limit Laws for Likelihood Ratios.- 5.1 Introduction.- 5.2 Auxiliary Results.- 5.2.1 Lindeberg's Procedure.- 5.2.2 Levy Splittings.- 5.2.3 Paul Levy's Symmetrization Inequalities.- 5.2.4 Conditions for Shift-Compactness.- 5.2.5 A Central Limit Theorem for Infinitesimal Arrays.- 5.2.6 The Special Case of Gaussian Limits.- 5.2.7 Peano Differentiable Functions.- 5.3 Limits for Binary Experiments.- 5.4 Gaussian Limits.- 5.5 Historical Remarks.- 6 Local Asymptotic Normality.- 6.1 Introduction.- 6.2 Locally Asymptotically Quadratic Families.- 6.3 A Method of Construction of Estimates.- 6.4 Some Local Bayes Properties.- 6.5 Invariance and Regularity.- 6.6 The LAMN and LAN Conditions.- 6.7 Additional Remarks on the LAN Conditions.- 6.8 Wald's Tests and Confidence Ellipsoids.- 6.9 Possible Extensions.- 6.10 Historical Remarks.- 7 Independent, Identically Distributed Observations.- 7.1 Introduction.- 7.2 The Standard i.i.d. Case: Differentiability in Quadratic Mean.- 7.3 Some Examples.- 7.4 Some Nonparametric Considerations.- 7.5 Bounds on the Risk of Estimates.- 7.6 Some Cases Where the Number of Observations Is Random.- 7.7 Historical Remarks.- 8 On Bayes Procedures.- 8.1 Introduction.- 8.2 Bayes Procedures Behave Nicely.- 8.3 The Bernstein-von Mises Phenomenon.- 8.4 A Bernstein-von Mises Result for the i.i.d. Case.- 8.5 Bayes Procedures Behave Miserably.- 8.6 Historical Remarks.- Author Index.

Asymptotics in Statistics–Some Basic Concepts

We focus on the problem of estimating the graph structure associated with a discrete Markov random field We describe a method based on l1-regularized logistic regression, in which the neighborhood of any given node is estimated by performing logistic regression subject to an l1-constraint Our framework applies to the high-dimensional setting, in which both the number of nodes p and maximum neighborhood sizes d are allowed to grow as a function of the number of observations n Our main result is to establish sufficient conditions on the triple (n, p, d) for the method to succeed in consistently estimating the neighborhood of every node in the graph simultaneously Under certain mutual incoherence conditions analogous to those imposed in previous work on linear regression, we prove that consistent neighborhood selection can be obtained as long as the number of observations n grows more quickly than 6d6 log d + 2d5 log p, thereby establishing that logarithmic growth in the number of samples n relative to graph size p is sufficient to achieve neighborhood consistency

High-Dimensional Graphical Model Selection Using ℓ1-Regularized Logistic Regression

An evolutionary multi-objective optimization tool based on an estimation of distribution algorithm is proposed. The algorithm uses the ranking method of non-dominated sorting genetic algorithm-II and the Parzen estimator to approximate the probability density of solutions lying on the Pareto front. The proposed algorithm has been applied to different types of test case problems and results show good performance of the overall optimization procedure in terms of the number of function evaluations. An alternative spreading technique that uses the Parzen estimator in the objective function space is proposed as well. When this technique is used, achieved results appear to be qualitatively equivalent to those previously obtained by adopting the crowding distance described in non-dominated sorting genetic algorithm-II.

MOPED: A multi-objective parzen-based estimation of distribution algorithm for continuous problems

This paper presents a hybrid evolutionary optimization strategy combining the Mixed Bayesian Optimization Algorithm (MBOA) with variance adaptation as implemented in Evolution Strategies. This new approach is intended to circumvent some of the deficiences of MBOA with unimodal functions and to enhance its adaptivity. The Adaptive MBOA algorithm - AMBOA - is compared with the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). The comparison shows that, in continuous domains, AMBOA is more efficient than the original MBOA algorithm and its performance on separable unimodal functions is comparable to that of CMA-ES.

A Mixed Bayesian Optimization Algorithm with variance adaptation

In evolutionary continuous optimization by building and using probabilistic models, the multivariate Gaussian distribution and their variants or extensions such as the mixture of Gaussians have been used popularly. However, this Gaussian assumption is often violated in many real problems. In this paper, we propose a new continuous estimation of distribution algorithms (EDAs) with the variational Bayesian independent component analyzers mixture model (vbICA-MM) for allowing any distribution to be modeled. We examine how this sophisticated density estimation technique has influence on the performance of the optimization by employing the same selection and population alternation schemes used in the previous EDAs. Our experimental results support that the presented EDAs achieve better performance than previous EDAs with ICA and Gaussian mixture- or kernel-based approaches.

Evolutionary continuous optimization by distribution estimation with variational Bayesian independent component analyzers mixture model

Probabilistic modeling is the de?ning characteristic of estimation of distribution algorithms (EDAs) which determines their behavior and performance in optimization. Regularization is a well-known statistical technique used for obtaining an improved model by reducing the generalization error of estimation, especially in high-dimensional problems. `1-regularization is a type of this technique with the appealing variable selection property which results in sparse model estimations. In this thesis, we study the use of regularization techniques for model learning in EDAs. Several methods for regularized model estimation in continuous domains based on a Gaussian distribution assumption are presented, and analyzed from di?erent aspects when used for optimization in a high-dimensional setting, where the population size of EDA has a logarithmic scale with respect to the number of variables. The optimization results obtained for a number of continuous problems with an increasing number of variables show that the proposed EDA based on regularized model estimation performs a more robust optimization, and is able to achieve signi?cantly better results for larger dimensions than other Gaussian-based EDAs. We also propose a method for learning a marginally factorized Gaussian Markov random ?eld model using regularization techniques and a clustering algorithm. The experimental results show notable optimization performance on continuous additively decomposable problems when using this model estimation method. Our study also covers multi-objective optimization and we propose joint probabilistic modeling of variables and objectives in EDAs based on Bayesian networks, speci?cally models inspired from multi-dimensional Bayesian network classi?ers. It is shown that with this approach to modeling, two new types of relationships are encoded in the estimated models in addition to the variable relationships captured in other EDAs: objectivevariable and objective-objective relationships. An extensive experimental study shows the e?ectiveness of this approach for multi- and many-objective optimization. With the proposed joint variable-objective modeling, in addition to the Pareto set approximation, the algorithm is also able to obtain an estimation of the multi-objective problem structure. Finally, the study of multi-objective optimization based on joint probabilistic modeling is extended to noisy domains, where the noise in objective values is represented by intervals. A new version of the Pareto dominance relation for ordering the solutions in these problems, namely ?-degree Pareto dominance, is introduced and its properties are analyzed. We show that the ranking methods based on this dominance relation can result in competitive performance of EDAs with respect to the quality of the approximated Pareto sets. This dominance relation is then used together with a method for joint probabilistic modeling based on `1-regularization for multi-objective feature subset selection in classi?cation, where six di?erent measures of accuracy are considered as objectives with interval values. The individual assessment of the proposed joint probabilistic modeling and solution ranking methods on datasets with small-medium dimensionality, when using two di?erent Bayesian classi?ers, shows that comparable or better Pareto sets of feature subsets are approximated in comparison to standard methods.

Hossein Karshenas

Papers

Regularized model learning in EDAs for continuous andmulti-objective optimization