In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/84479E2B03A9BFFE0F9CD46CF9FCD289/S0962492919000060a.pdf/div-class-title-derivative-free-optimization-methods-div.pdf

Derivative-free optimization methods

Robust principal component analysis (RPCA) via decomposition into low-rank plus sparse matrices offers a powerful framework for a large variety of applications such as image processing, video processing, and 3-D computer vision. Indeed, most of the time these applications require to detect sparse outliers from the observed imagery data that can be approximated by a low-rank matrix. Moreover, most of the time experiments show that RPCA with additional spatial and/or temporal constraints often outperforms the state-of-the-art algorithms in these applications. Thus, the aim of this paper is to survey the applications of RPCA in computer vision. In the first part of this paper, we review representative image processing applications as follows: 1) low-level imaging such as image recovery and denoising, image composition, image colorization, image alignment and rectification, multifocus image, and face recognition; 2) medical imaging such as dynamic magnetic resonance imaging (MRI) for acceleration of data acquisition, background suppression, and learning of interframe motion fields; and 3) imaging for 3-D computer vision with additional depth information such as in structure from motion (SfM) and 3-D motion recovery. In the second part, we present the applications of RPCA in video processing which utilize additional spatial and temporal information compared to image processing. Specifically, we investigate video denoising and restoration, hyperspectral video, and background/foreground separation. Finally, we provide perspectives on possible future research directions and algorithmic frameworks that are suitable for these applications.

/pdf/on-the-applications-of-robust-pca-in-image-and-video-51546ljg1c.pdf

On the Applications of Robust PCA in Image and Video Processing

A conjugate gradient method to solve convex constrained monotone equations with applications in compressive sensing

To reduce the communication among processors and improve the computing time for solving linear complementarity problems, we present a two-step modulus-based synchronous multisplitting iteration method and the corresponding symmetric modulus-based multisplitting relaxation methods. The convergence theorems are established when the system matrix is an H+-matrix, which improve the existing convergence theory. Numerical results show that the symmetric modulus-based multisplitting relaxation methods are effective in actual implementation.

Modulus‐based synchronous multisplitting iteration methods for linear complementarity problems

This paper considers optimization problems on the Stiefel manifold $$X^{\mathsf{T}}X=I_p$$XTX=Ip, where $$X\in \mathbb {R}^{n \times p}$$X?Rn×p is the variable and $$I_p$$Ip is the $$p$$p-by-$$p$$p identity matrix. A framework of constraint preserving update schemes is proposed by decomposing each feasible point into the range space of $$X$$X and the null space of $$X^{\mathsf{T}}$$XT. While this general framework can unify many existing schemes, a new update scheme with low complexity cost is also discovered. Then we study a feasible Barzilai---Borwein-like method under the new update scheme. The global convergence of the method is established with an adaptive nonmonotone line search. The numerical tests on the nearest low-rank correlation matrix problem, the Kohn---Sham total energy minimization and a specific problem from statistics demonstrate the efficiency of the new method. In particular, the new method performs remarkably well for the nearest low-rank correlation matrix problem in terms of speed and solution quality and is considerably competitive with the widely used SCF iteration for the Kohn---Sham total energy minimization.

http://www.optimization-online.org/DB_FILE//2013/01/3721.pdf

A framework of constraint preserving update schemes for optimization on Stiefel manifold

A class of derivative-free methods for large-scale nonlinear monotone equations

Based on the well-known result that the sum of the largest eigenvalues of a symmetric matrix can be represented as a semidefinite programming problem (SDP), we formulate the nearest low-rank correlation matrix problem as a nonconvex SDP and propose a numerical method that solves a sequence of least-square problems. Each of the least-square problems can be solved by a specifically designed semismooth Newton method, which is shown to be quadratically convergent. The sequential method is guaranteed to produce a stationary point of the nonconvex SDP. Our numerical results demonstrate the high efficiency of the proposed method on large scale problems.

A Sequential Semismooth Newton Method for the Nearest Low-rank Correlation Matrix Problem

Support vector machine is an important and fundamental technique in machine learning. In this paper, we apply a semismooth Newton method to solve two typical SVM models: the L2-loss SVC model and the \epsilon-L2-loss SVR model. The semismooth Newton method is widely used in optimization community. A common belief on the semismooth Newton method is its fast convergence rate as well as high computational complexity. Our contribution in this paper is that by exploring the sparse structure of the models, we significantly reduce the computational complexity, meanwhile keeping the quadratic convergence rate. Extensive numerical experiments demonstrate the outstanding performance of the semismooth Newton method, especially for problems with huge size of sample data (for news20.binary problem with 19996 features and 1355191 samples, it only takes three seconds). In particular, for the \epsilon-L2-loss SVR model, the semismooth Newton method significantly outperforms the leading solvers including DCD and TRON.

A Semismooth Newton Method for Support Vector Classification and Regression

Support vector machine is an important and fundamental technique in machine learning. In this paper, we apply a semismooth Newton method to solve two typical SVM models: the L2-loss SVC model and the $$\epsilon $$
 -L2-loss SVR model. The semismooth Newton method is widely used in optimization community. A common belief on the semismooth Newton method is its fast convergence rate as well as high computational complexity. Our contribution in this paper is that by exploring the sparse structure of the models, we significantly reduce the computational complexity, meanwhile keeping the quadratic convergence rate. Extensive numerical experiments demonstrate the outstanding performance of the semismooth Newton method, especially for problems with huge size of sample data (for news20.binary problem with 19,996 features and 1,355,191 samples, it only takes 3 s). In particular, for the $$\epsilon $$
 -L2-loss SVR model, the semismooth Newton method significantly outperforms the leading solvers including DCD and TRON.

A semismooth Newton method for support vector classification and regression

Robust PCA has found important applications in many areas, such as video surveillance, face recognition, latent semantic indexing and so on. In this paper, we study its application in ground moving target indication (GMTI) in wide-area surveillance radar system. MTI is the key task in wide-area surveillance radar system. Due to its great importance in future reconnaissance systems, it attracts great interest from scientists. In (Yan et al. in IEEE Geosci. Remote Sens. Lett., 10:617–621, 2013), the authors first introduced robust PCA to model the GMTI problem, and demonstrate promising simulation results to verify the advantages over other models. However, the robust PCA model can not fully describe the problem. As pointed out in (Yan et al. in IEEE Geosci. Remote Sens. Lett., 10:617–621, 2013), due to the special structure of the sparse matrix (which includes the moving target information), there will be difficulties for the exact extraction of moving targets. This motivates our work in this paper where we will detail the GMTI problem, explore the mathematical properties and discuss how to set up better models to solve the problem. We propose two models, the structured RPCA model and the row-modulus RPCA model, both of which will better fit the problem and take more use of the special structure of the sparse matrix. Simulation results confirm the improvement of the proposed models over the one in (Yan et al. in IEEE Geosci. Remote Sens. Lett., 10:617–621, 2013).

https://link.springer.com/content/pdf/10.1007%2Fs40305-013-0006-y.pdf

Qingna Li

Papers

A class of derivative-free methods for large-scale nonlinear monotone equations

A Sequential Semismooth Newton Method for the Nearest Low-rank Correlation Matrix Problem

A Semismooth Newton Method for Support Vector Classification and Regression

A semismooth Newton method for support vector classification and regression

Robust PCA for Ground Moving Target Indication in Wide-Area Surveillance Radar System