We consider ``one-at-a-time'' coordinate-wise descent algorithms for a class of convex optimization problems. An algorithm of this kind has been proposed for the $L_1$-penalized regression (lasso) in the literature, but it seems to have been largely ignored. Indeed, it seems that coordinate-wise algorithms are not often used in convex optimization. We show that this algorithm is very competitive with the well-known LARS (or homotopy) procedure in large lasso problems, and that it can be applied to related methods such as the garotte and elastic net. It turns out that coordinate-wise descent does not work in the ``fused lasso,'' however, so we derive a generalized algorithm that yields the solution in much less time that a standard convex optimizer. Finally, we generalize the procedure to the two-dimensional fused lasso, and demonstrate its performance on some image smoothing problems.

Pathwise coordinate optimization

We consider “one-at-a-time” coordinate-wise descent algorithms for a class of convex optimization problems. An algorithm of this kind has been proposed for the L1-penalized regression (lasso) in the literature, but it seems to have been largely ignored. Indeed, it seems that coordinate-wise algorithms are not often used in convex optimization. We show that this algorithm is very competitive with the well-known LARS (or homotopy) procedure in large lasso problems, and that it can be applied to related methods such as the garotte and elastic net. It turns out that coordinate-wise descent does not work in the “fused lasso,” however, so we derive a generalized algorithm that yields the solution in much less time that a standard convex optimizer. Finally, we generalize the procedure to the two-dimensional fused lasso, and demonstrate its performance on some image smoothing problems. 1. Introduction. In this paper we consider statistical models that lead to convex optimization problems with inequality constraints. Typically, the optimization for these problems is carried out using a standard quadratic programming algorithm. The purpose of this paper is to explore “one-at-a-time” coordinate-wise descent algorithms for these problems. The equivalent of a coordinate descent algorithm has been proposed for the L1-penalized regression (lasso) in the literature, but it is not commonly used. Moreover, coordinate-wise algorithms seem too simple, and they are not often used in convex optimization, perhaps because they only work in specialized problems. We ourselves never appreciated the value of coordinate descent methods for convex statistical problems before working on this paper. In this paper we show that coordinate descent is very competitive with the wellknown LARS (or homotopy) procedure in large lasso problems, can deliver a path of solutions efficiently, and can be applied to many other convex statistical problems such as the garotte and elastic net. We then go on to explore a nonseparable problem in which coordinate-wise descent does not work—the “fused lasso.” We derive a generalized algorithm that yields the solution in much less time that a standard convex optimizer. Finally, we generalize the procedure to

/pdf/pathwise-coordinate-optimization-4li14fejub.pdf

Imposition of a lasso penalty shrinks parameter estimates toward zero and performs continuous model selection. Lasso penalized regression is capable of handling linear regression problems where the number of predictors far exceeds the number of cases. This paper tests two exceptionally fast algorithms for estimating regression coefficients with a lasso penalty. The previously known l2 algorithm is based on cyclic coordinate descent. Our new l1 algorithm is based on greedy coordinate descent and Edgeworth’s algorithm for ordinary l1 regression. Each algorithm relies on a tuning constant that can be chosen by cross-validation. In some regression problems it is natural to group parameters and penalize parameters group by group rather than separately. If the group penalty is proportional to the Euclidean norm of the parameters of the group, then it is possible to majorize the norm and reduce parameter estimation to l2 regression with a lasso penalty. Thus, the existing algorithm can be extended to novel settings. Each of the algorithms discussed is tested via either simulated or real data or both. The Appendix proves that a greedy form of the l2 algorithm converges to the minimum value of the objective function.

/pdf/coordinate-descent-algorithms-for-lasso-penalized-regression-5bld7rvrz4.pdf

Coordinate descent algorithms for lasso penalized regression

A major challenge in real-world feature matching problems is to tolerate the numerous outliers arising in typical visual tasks. Variations in object appearance, shape, and structure within the same object class make it harder to distinguish inliers from outliers due to clutters. In this paper, we propose a max-pooling approach to graph matching, which is not only resilient to deformations but also remarkably tolerant to outliers. The proposed algorithm evaluates each candidate match using its most promising neighbors, and gradually propagates the corresponding scores to update the neighbors. As final output, it assigns a reliable score to each match together with its supporting neighbors, thus providing contextual information for further verification. We demonstrate the robustness and utility of our method with synthetic and real image experiments.

/pdf/finding-matches-in-a-haystack-a-max-pooling-strategy-for-1xjdrb3wdt.pdf

Finding Matches in a Haystack: A Max-Pooling Strategy for Graph Matching in the Presence of Outliers

Recent results in compressed sensing show that a sparse or compressible signal can be reconstructed from a few incoherent measurements. Since noise is always present in practical data acquisition systems, sensing, and reconstruction methods are developed assuming a Gaussian (light-tailed) model for the corrupting noise. However, when the underlying signal and/or the measurements are corrupted by impulsive noise, commonly employed linear sampling operators, coupled with current reconstruction algorithms, fail to recover a close approximation of the signal. In this paper, we propose robust methods for sampling and reconstructing sparse signals in the presence of impulsive noise. To solve the problem of impulsive noise embedded in the underlying signal prior the measurement process, we propose a robust nonlinear measurement operator based on the weighed myriad estimator. In addition, we introduce a geometric optimization problem based on L 1 minimization employing a Lorentzian norm constraint on the residual error to recover sparse signals from noisy measurements. Analysis of the proposed methods show that in impulsive environments when the noise posses infinite variance we have a finite reconstruction error and furthermore these methods yield successful reconstruction of the desired signal. Simulations demonstrate that the proposed methods significantly outperform commonly employed compressed sensing sampling and reconstruction techniques in impulsive environments, while providing comparable performance in less demanding, light-tailed environments.

Robust Sampling and Reconstruction Methods for Sparse Signals in the Presence of Impulsive Noise

Least absolute deviation (LAD) regression is an important tool used in numerous applications throughout science and engineering, mainly due to the intrinsic robust characteristics of LAD In this paper, we show that the optimization needed to solve the LAD regression problem can be viewed as a sequence of maximum likelihood estimates (MLE) of location The derived algorithm reduces to an iterative procedure where a simple coordinate transformation is applied during each iteration to direct the optimization procedure along edge lines of the cost surface, followed by an MLE of location which is executed by a weighted median operation Requiring weighted medians only, the new algorithm can be easily modularized for hardware implementation, as opposed to most of the other existing LAD methods which require complicated operations such as matrix entry manipulations One exception is Wesolowsky's direct descent algorithm, which among the top algorithms is also based on weighted median operations Simulation shows that the new algorithm is superior in speed to Wesolowsky's algorithm, which is simple in structure as well The new algorithm provides a better tradeoff solution between convergence speed and implementation complexity

/pdf/a-maximum-likelihood-approach-to-least-absolute-deviation-1qit3f5esl.pdf

A maximum likelihood approach to least absolute deviation regression

Summary form only given. Multi-channel and multi-spectral signals are often correlated across channels. Moreover, multispectral images have considerable similarity in in-channel correlations. But in array signal processing, due to the existence of multiple frequency components and their phase shifts, in-channel correlations may vary drastically. In this paper, a new multichannel weighted median filter is proposed which can capture the general correlation structure in array signals and process them in an efficient manner. The algorithm is further extended onto the complex domain by means of "phase coupling". The performance of the filter is presented in a three-sensor array processing example.

http://ieeexplore.ieee.org/iel5/10019/32163/01502297.pdf

A multichannel weighted median filter for complex array signal processing

We show that the maximum likelihood (ML) estimate of location under the Laplacian model, which forms the basis for weighted median filters, can be generalized to correlation estimates based on weighted medians. Much like linear sample correlations, the resultant median correlation estimates have a surprisingly simple structure. Unlike linear correlations, median correlations are robust to data contamination. Notably, weights in this framework do not assume fixed values as with weighted median filters but take on random values determined by the underlying data itself. The underlying parameters associated with the sample median correlations are obtained, leading to well-defined expressions that can be used in subspace-based signal processing algorithms. The properties of median correlations are illustrated through a number of simulations where the MUltiple SIgnal Classification (MUSIC) algorithm is applied on linear and median sample correlation matrices for real-valued frequency estimation applications. This paper thus unveils new and powerful capabilities of weighted medians for use in modern signal processing applications.

Median power and median correlation theory

Weighted medians over multichannel signals are not uniquely defined. Due to its simplicity, Astola 's Vector Median (VM) has received considerable attention particularly in image processing applications. In this paper, we show that the VM and its direct extension the Weighted VM are limited as they do not fully utilize the cross-channel correlation. In fact, VM treats all sub-channel components independent of each other. By revisiting the principles of Maximum Likelihood estimation of location in a multivariate signal space, we propose two new and conceptually simple multichannel weighted median filters which can capture cross-channel information effectively. Their optimal filter derivations are also presented, followed by a series of simulations from color image denoising to array signal processing where the advantages of the new filtering structures are illustrated

Weighted Median Filters for Multichannel Signals

Inspired by the maximum likelihood (ML) estimates of location in multivariate spaces, we introduce in this paper a new filtering structure capable of capturing and exploiting both spatial and cross-channel correlations embedded in the data. An adaptive optimization algorithm for a sub-optimal realization of the proposed generalized vector median (GVM) filter, namely the marginal GVM, is derived. The effectiveness of the algorithm is shown through a color image denoising experiment.

Yinbo Li

Papers

A maximum likelihood approach to least absolute deviation regression

A multichannel weighted median filter for complex array signal processing

Median power and median correlation theory

Weighted Median Filters for Multichannel Signals

Generalized vector medians for correlated channels