Weak Convergence and Empirical Processes - A. W. van der Vaart; J. A. Wellner.

Universality of deep convolutional neural networks

Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, and many other domains. The involved deep neural network architectures and computational issues have been well studied in machine learning. But there lacks a theoretical foundation for understanding the approximation or generalization ability of deep learning methods generated by the network architectures such as deep convolutional neural networks having convolutional structures. Here we show that a deep convolutional neural network (CNN) is universal, meaning that it can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough. This answers an open question in learning theory. Our quantitative estimate, given tightly in terms of the number of free parameters to be computed, verifies the efficiency of deep CNNs in dealing with large dimensional data. Our study also demonstrates the role of convolutions in deep CNNs.

Universality of Deep Convolutional Neural Networks

Mapping tropical tree species at landscape scales to provide information for ecologists and forest managers is a new challenge for the remote sensing community. For this purpose, detection and delineation of individual tree crowns (ITCs) is a prerequisite. Here, we present a new method of automatic tree crown delineation based only on very high resolution images from WorldView-2 satellite and apply it to a region of the Atlantic rain forest with highly heterogeneous tropical canopy cover – the Santa Genebra forest reserve in Brazil. The method works in successive steps that involve pre-processing, selection of forested pixels, enhancement of borders, detection of pixels in the crown borders, correction of shade in large trees and, finally, segmentation of the tree crowns. Principally, the method uses four techniques: rolling ball algorithm and mathematical morphological operations to enhance the crown borders and ease the extraction of tree crowns; bimodal distribution parameters estimations to identify the shaded pixels in the gaps, borders, and crowns; and focal statistics for the analysis of neighbouring pixels. Crown detection is validated by comparing the delineated ITCs with a sample of ITCs delineated manually by visual interpretation. In addition, to test if the spectra of individual species are conserved in the automatic delineated crowns, we compare the accuracy of species prediction with automatic and manual delineated crowns with known species. We find that our method permits detection of up to 80% of ITCs. The seven species with over 10 crowns identified in the field were mapped with reasonable accuracy (30.5–96%) given that only WorldView-2 bands and texture features were used. Similar classification accuracies were obtained using both automatic and manual delineation, thereby confirming that species’ spectral responses are preserved in the automatic method and thus permitting the recognition of species at the landscape scale. Our method might support tropical forest applications, such as mapping species and canopy characteristics at the landscape scale.

Individual tree crown delineation in a highly diverse tropical forest using very high resolution satellite images

In this paper, we propose a comprehensive analytics framework that can serve as a decision support tool for HR recruiters in real-world settings in order to improve hiring and placement decisions. The proposed framework follows two main phases: a local prediction scheme for recruitments' success at the level of a single job placement, and a mathematical model that provides a global recruitment optimization scheme for the organization, taking into account multilevel considerations. In the first phase, a key property of the proposed prediction approach is the interpretability of the machine learning (ML) model, which in this case is obtained by applying the Variable-Order Bayesian Network (VOBN) model to the recruitment data. Specifically, we used a uniquely large dataset that contains recruitment records of hundreds of thousands of employees over a decade and represents a wide range of heterogeneous populations. Our analysis shows that the VOBN model can provide both high accuracy and interpretability insights to HR professionals. Moreover, we show that using the interpretable VOBN can lead to unexpected and sometimes counter-intuitive insights that might otherwise be overlooked by recruiters who rely on conventional methods. We demonstrate that it is feasible to predict the successful placement of a candidate in a specific position at a pre-hire stage and utilize predictions to devise a global optimization model. Our results show that in comparison to actual recruitment decisions, the devised framework is capable of providing a balanced recruitment plan while improving both diversity and recruitment success rates, despite the inherent trade-off between the two.

/pdf/employees-recruitment-a-prescriptive-analytics-approach-via-g3o5ey0ixu.pdf

Employees recruitment: A prescriptive analytics approach via machine learning and mathematical programming.

liquidSVM is a package written in C++ that provides SVM-type solvers for various classification and regression tasks. Because of a fully integrated hyper-parameter selection, very carefully implemented solvers, multi-threading and GPU support, and several built-in data decomposition strategies it provides unprecedented speed for small training sizes as well as for data sets of tens of millions of samples. Besides the C++ API and a command line interface, bindings to R, MATLAB, Java, Python, and Spark are available. We present a brief description of the package and report experimental comparisons to other SVM packages.

liquidSVM: A Fast and Versatile SVM package

We investigate iterated compositions of weighted sums of Gaussian kernels and provide an interpretation of the construction that shows some similarities with the architectures of deep neural networks. On the theoretical side, we show that these kernels are universal and that SVMs using these kernels are universally consistent. We further describe a parameter optimization method for the kernel parameters and empirically compare this method to SVMs, random forests, a multiple kernel learning approach, and to some deep neural networks.

Learning with Hierarchical Gaussian Kernels.

We perform simulations for one dimensional continuous-time random walks in two dynamic random environments with fast (independent spin-flips) and slow (simple symmetric exclusion) decay of space-time correlations, respectively. We focus on the asymptotic speeds and the scaling limits of such random walks. We observe different behaviors depending on the dynamics of the underlying random environment and the ratio between the jump rate of the random walk and the one of the environment. We compare our data with well known results for static random environment. We observe that the non-diffusive regime known so far only for the static case can occur in the dynamical setup too. Such anomalous fluctuations give rise to a new phase diagram. Further we discuss possible consequences for more general static and dynamic random environments.

/pdf/continuity-and-anomalous-fluctuations-in-random-walks-in-3jnz5tzz7x.pdf

Continuity and Anomalous Fluctuations in Random Walks in Dynamic Random Environments: Numerics, Phase Diagrams and Conjectures

We perform simulations for one dimensional continuous-time random walks in two dynamic random environments with fast (independent spin-flips) and slow (simple symmetric exclusion) decay of space-time correlations, respectively. We focus on the asymptotic speeds and the scaling limits of such random walks. We observe different behaviors depending on the dynamics of the underlying random environment and the ratio between the jump rate of the random walk and the one of the environment. We compare our data with well known results for static random environment. We observe that the non-diffusive regime known so far only for the static case can occur in the dynamic setup too. Such anomalous fluctuations give rise to a new phase diagram. Further we discuss possible consequences for more general static and dynamic random environments.

Continuity and anomalous fluctuations in random walks in dynamic random environments: numerics, phase diagrams and conjectures

Although support vector machines (SVMs) are theoretically well understood, their underlying optimization problem becomes very expensive, if, for example, hundreds of thousands of samples and a non-linear kernel are considered. Several approaches have been proposed in the past to address this serious limitation. In this work we investigate a decomposition strategy that learns on small, spatially defined data chunks. Our contributions are two fold: On the theoretical side we establish an oracle inequality for the overall learning method using the hinge loss, and show that the resulting rates match those known for SVMs solving the complete optimization problem with Gaussian kernels. On the practical side we compare our approach to learning SVMs on small, randomly chosen chunks. Here it turns out that for comparable training times our approach is significantly faster during testing and also reduces the test error in most cases significantly. Furthermore, we show that our approach easily scales up to 10 million training samples: including hyper-parameter selection using cross validation, the entire training only takes a few hours on a single machine. Finally, we report an experiment on 32 million training samples. All experiments used liquidSVM (Steinwart and Thomann, 2017).

/pdf/spatial-decompositions-for-large-scale-svms-5q4bcspx8m.pdf

Philipp Thomann

Papers

liquidSVM: A Fast and Versatile SVM package

Learning with Hierarchical Gaussian Kernels.

Continuity and Anomalous Fluctuations in Random Walks in Dynamic Random Environments: Numerics, Phase Diagrams and Conjectures

Continuity and anomalous fluctuations in random walks in dynamic random environments: numerics, phase diagrams and conjectures

Spatial Decompositions for Large Scale SVMs