Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

[신간의 별자리x] 우리/미술, 그리고 ‘슬픔의 박물관’

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Data Mining - Concepts and Techniques.

In this article, we propose a light detection and ranging (LiDAR) data denoising scheme for wind profile observation as a part of quality control procedure for wind velocity monitoring and windshear detection. +e proposed denoising scheme consists of several components. (i) It selects LiDAR observations according to their SNR values so that serious noisy data can be removed. (ii) A polar-based total variation smoothing term is employed to regularize LiDAR observations. (iii) +e regularization parameters are automatically determined to balance the data-fitting term and the total variation smoothing term. Numerical results for LiDAR data collected at the Hong Kong International Airport are reported to demonstrate that the denoising performance of the proposed method is better than that of the testing LiDAR data denoising schemes in the literature.

Observation Selection, Total Variation, and L-Curve Methods for LiDAR Data Denoising

The performance of deep learning-based denoisers highly depends on the quantity and quality of training data. However, paired noisy-clean training images are generally unavailable in hyperspectral remote sensing areas. To solve this problem, this work resorts to the self-supervised learning technique, where our proposed model can train itself to learn one part of noisy input from another part of noisy input. We study a general hyperspectral image (HSI) denoising framework, called Eigenimage2Eigenimage (E2E), which turns the HSI denoising problem into an eigenimage (i.e., the subspace representation coefficients of the HSI) denoising problem and proposes a learning strategy to generate noisy-noisy paired training eigenimages from noisy eigenimages. Consequently, the E2E denoising framework can be trained without clean data and applied to denoise HSIs without the constraint with the number of frequency bands. Experimental results are provided to demonstrate the performance of the proposed method that is better than the other existing deep learning methods for denoising HSIs. A MATLAB demo of this work is available at https://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimagehttps://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimage for the sake of reproducibility.

Eigenimage2Eigenimage (E2E): A Self-Supervised Deep Learning Network for Hyperspectral Image Denoising.

Multiple loci analysis has become popular with the advanced development in biological experiments. A lot of studies have been focused on the biological and the statistical properties of such multiple loci analysis. In this paper, we study one of the important computational problems: solving the probabilities of haplotype classes from a large linear system Ax = b derived from the recombination events in multiple loci analysis. Since the size of the recombination matrix A increases exponentially with respect to the number of loci, fast solvers are required to deal with a large number of loci in the analysis. By exploiting the nice structure of the matrix A, we develop an efficient recursive algorithm for solving such structured linear systems. In particular, the complexity of the proposed algorithm is of O(mlogm) operations and the memory requirement is of O(m) locations where m is the size of the matrix A. Numerical examples are given to demonstrate the effectiveness of our efficient solver.

/pdf/a-recursive-method-for-solving-haplotype-frequencies-in-52aaf5zavk.pdf

A recursive method for solving haplotype frequencies in multiple loci linkage analysis

Heterogeneous Information Network(HIN) collecitve classification studies the problem of predicting labels for one type of nodes in a HIN which contains multiple types of nodes multiple types of links among them. Previous studies have revealed that exploiting relative importance of links is quite useful to improve node classification performance as connected nodes tend to have similar labels. Most existing approaches exploit the relative importance of links either by directly counting the number of connections among nodes or by learning the weight of each type of link from labeled data only. However, these approaches either neglect the importance of types of links to the class labels or may lead to overfitting problem. We propose a Tensor-based Markov chain (T-Mark) approach, which is able to automatically and simultaneously predict the labels for unlabeled nodes and give the relative importance of types of links that actually improve the classification accuracy. Specifically, we build two tensor equations by using the HIN and features of nodes from both labeled and unlabeled data. A Markov chain-based model is proposed and it is solved by an iterative process to obtain the stationary distributions. Theoretical analyses of the existence and uniqueness of such probability distributions are given. Extensive experimental results demonstrate that T-Mark is able to achieve superior performance in the comparison and obtain reasonable relative importance of links.

A Tensor-based Markov Chain Model for Heterogeneous Information Network Collective Classification

Windshear is a kind of microscale meteorological phenomenon which can cause danger to the landing and takeoff of aircrafts. Accurate windshear detection plays a crucial role in aviation safety. With the development of machine learning, several learning-based methods are proposed for windshear detection, i.e., windshear and non-windshear classification. To obtain accurate detection results, it is significant to extract features that can distinguish windshear and non-windshear properly from the obtained wind velocity data. In this paper, we mainly introduce two statistical indicators derived from the Doppler Light Detection and Ranging (LiDAR) observational wind velocity data by plan position illustrate (PPI) scans for windshear features construction. Besides the indicators directly derived from the wind velocity data, we also study the visual information from the corresponding conical images of wind velocity. Based on the proposed indicators, we construct three feature vectors for windshear and non-windshear classification. Inspired by the idea of multiple instance learning, the wind velocity data collected in the 4 minutes within the reported time spot are considered in the procedure of feature vector construction, which can reduce the possibility of windshear features missing. Both statistical methods and clustering methods are applied to evaluate the effectiveness of the proposed feature vectors. Numerical results show that the proposed feature vectors have good effect on windshear and non-windshear classification and can be used to provide more accurate windshear alerting to pilots in practice.

/pdf/lidar-based-windshear-detection-via-statistical-features-2tyvxwaw.pdf

Michael K. Ng

Papers

Observation Selection, Total Variation, and L-Curve Methods for LiDAR Data Denoising

Eigenimage2Eigenimage (E2E): A Self-Supervised Deep Learning Network for Hyperspectral Image Denoising.

A recursive method for solving haplotype frequencies in multiple loci linkage analysis

A Tensor-based Markov Chain Model for Heterogeneous Information Network Collective Classification

LiDAR-Based Windshear Detection via Statistical Features