On robust estimation of the location parameter

Order Restricted Statistical Inference.

We propose a new method of effective dimension reduction for a multiindex model which is based on iterative improvement of the family of average derivative estimates. The procedure is computationally straightforward and does not require any prior information about the structure of the underlying model. We show that in the case when the effective dimension m of the index space does not exceed 3, this space can be estimated with the rate n −1/2 under rather mild assumptions on the model.

Structure Adaptative Approach for Dimension Reduction

Real world data often exhibit low-dimensional geometric structures, and can
be viewed as samples near a low-dimensional manifold. This paper studies
nonparametric regression of H\"older functions on low-dimensional manifolds
using deep ReLU networks. Suppose $n$ training data are sampled from a H\"older
function in $\mathcal{H}^{s,\alpha}$ supported on a $d$-dimensional Riemannian
manifold isometrically embedded in $\mathbb{R}^D$, with sub-gaussian noise. A
deep ReLU network architecture is designed to estimate the underlying function
from the training data. The mean squared error of the empirical estimator is
proved to converge in the order of $n^{-\frac{2(s+\alpha)}{2(s+\alpha) +
d}}\log^3 n$. This result shows that deep ReLU networks give rise to a fast
convergence rate depending on the data intrinsic dimension $d$, which is
usually much smaller than the ambient dimension $D$. It therefore demonstrates
the adaptivity of deep ReLU networks to low-dimensional geometric structures of
data, and partially explains the power of deep ReLU networks in tackling
high-dimensional data with low-dimensional geometric structures.

Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks

We investigate the implementation of a new stochastic Kuramoto-Vicsek-type model for global optimization of nonconvex functions on the sphere. This model belongs to the class of Consensus-Based Optimization. In fact, particles move on the sphere driven by a drift towards an instantaneous consensus point, which is computed as a convex combination of particle locations, weighted by the cost function according to Laplace's principle, and it represents an approximation to a global minimizer. The dynamics is further perturbed by a random vector field to favor exploration, whose variance is a function of the distance of the particles to the consensus point. In particular, as soon as the consensus is reached the stochastic component vanishes. The main results of this paper are about the proof of convergence of the numerical scheme to global minimizers provided conditions of well-preparation of the initial datum. The proof combines previous results of mean-field limit with a novel asymptotic analysis, and classical convergence results of numerical methods for SDE. We present several numerical experiments, which show that the algorithm proposed in the present paper scales well with the dimension and is extremely versatile. To quantify the performances of the new approach, we show that the algorithm is able to perform essentially as good as ad hoc state of the art methods in challenging problems in signal processing and machine learning, namely the phase retrieval problem and the robust subspace detection.

Consensus-Based Optimization on the Sphere: Convergence to Global Minimizers and Machine Learning

We study the approximation of two-layer compositions $f(x) = g(\phi(x))$ via deep ReLU networks, where $\phi$ is a nonlinear, geometrically intuitive, and dimensionality reducing feature map. We focus on two complementary choices for $\phi$ that are intuitive and frequently appearing in the statistical literature. The resulting approximation rates are near optimal and show adaptivity to intrinsic notions of complexity, which significantly extend a series of recent works on approximating targets over low-dimensional manifolds. Specifically, we show that ReLU nets can express functions, which are invariant to the input up to an orthogonal projection onto a low-dimensional manifold, with the same efficiency as if the target domain would be the manifold itself. This implies approximation via ReLU nets is faithful to an intrinsic dimensionality governed by the target $f$ itself, rather than the dimensionality of the approximation domain. As an application of our approximation bounds, we study empirical risk minimization over a space of sparsely constrained ReLU nets under the assumption that the conditional expectation satisfies one of the proposed models. We show near-optimal estimation guarantees in regression and classifications problems, for which, to the best of our knowledge, no efficient estimator has been developed so far.

ReLU nets adapt to intrinsic dimensionality beyond the target domain.

We address the structure identification and the uniform approximation of two fully nonlinear layer neural networks of the type $f(x)=1^T h(B^T g(A^T x))$ on $\mathbb R^d$ from a small number of query samples. We approach the problem by sampling actively finite difference approximations to Hessians of the network. Gathering several approximate Hessians allows reliably to approximate the matrix subspace $\mathcal W$ spanned by symmetric tensors $a_1 \otimes a_1 ,\dots,a_{m_0}\otimes a_{m_0}$ formed by weights of the first layer together with the entangled symmetric tensors $v_1 \otimes v_1 ,\dots,v_{m_1}\otimes v_{m_1}$, formed by suitable combinations of the weights of the first and second layer as $v_\ell=A G_0 b_\ell/\|A G_0 b_\ell\|_2$, $\ell \in [m_1]$, for a diagonal matrix $G_0$ depending on the activation functions of the first layer. The identification of the 1-rank symmetric tensors within $\mathcal W$ is then performed by the solution of a robust nonlinear program. We provide guarantees of stable recovery under a posteriori verifiable conditions. We further address the correct attribution of approximate weights to the first or second layer. By using a suitably adapted gradient descent iteration, it is possible then to estimate, up to intrinsic symmetries, the shifts of the activations functions of the first layer and compute exactly the matrix $G_0$. Our method of identification of the weights of the network is fully constructive, with quantifiable sample complexity, and therefore contributes to dwindle the black-box nature of the network training phase. We corroborate our theoretical results by extensive numerical experiments.

Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks

Adaptive multi-penalty regularization based on a generalized Lasso path

In this paper we study consensus-based optimization (CBO), which is a multi-agent metaheuristic derivative-free optimization method that can globally minimize nonconvex nonsmooth functions and is amenable to theoretical analysis. Based on an experimentally supported intuition that CBO always performs a gradient descent of the squared Euclidean distance to the global minimizer, we derive a novel technique for proving the convergence to the global minimizer in mean-field law for a rich class of objective functions. The result unveils internal mechanisms of CBO that are responsible for the success of the method. In particular, we prove that CBO performs a convexification of a very large class of optimization problems as the number of optimizing agents goes to infinity. Furthermore, we improve prior analyses by requiring minimal assumptions about the initialization of the method and by covering objectives that are merely locally Lipschitz continuous. As a by-product of the analysis, we establish a quantitative nonasymptotic Laplace principle, which may be of independent interest.

Timo Klock

Papers

ReLU nets adapt to intrinsic dimensionality beyond the target domain.

Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks

Adaptive multi-penalty regularization based on a generalized Lasso path

Consensus-based optimization methods converge globally in mean-field law.

Managing the microvibration impact on satellite performances